---
title: 'Intra-Cluster Mixup (ICM): Augmentation for CLL'
url: https://www.emergentmind.com/topics/intra-cluster-mixup-icm
type: topic
---

# Intra-Cluster Mixup (ICM): Augmentation for CLL

Intra-Cluster Mixup (ICM) is a data-augmentation framework tailored to complementary-label learning (CLL) that preserves the CLL constraint while leveraging label sharing, thereby reducing the complementary-label noise that makes vanilla Mixup ineffective in CLL. In CLL, training labels specify classes to which an instance does not belong rather than its ordinary class. The formulation introduced in "Intra-Cluster Mixup: An Effective Data Augmentation Technique for Complementary-Label Learning" treats the core obstacle for Mixup in this regime as a label-mismatch problem: naively mixing complementary labels can allocate probability mass to a true class of a mixed input, violating the complementary-label constraint. ICM addresses this mismatch by restricting mixing to nearby examples in a learned feature space, and is reported to improve balanced and imbalanced CLL across synthetic and real-world complementary-label datasets, with substantial gains on MNIST- and CIFAR-family benchmarks [2509.17971].

## 1. Complementary-label learning setting

CLL is a specialized form of weakly-supervised learning in which the learner observes only complementary labels. Let $K \ge 3$ denote the number of classes, and let the training set be
$$
\bar D = \{(x_i,\bar y_i)\}_{i=1}^N,\qquad x_i \in \mathbb{R}^d,\quad \bar y_i \in [K]\setminus\{y_i\},
$$
where $y_i$ is the unknown ordinary label. The goal is still ordinary classification at test time, with classifier
$$
f(x)=\arg\max_k g(x)_k.
$$
A standard loss-based CLL objective minimizes a surrogate complementary loss over the complementary dataset:
$$
\mathcal{L}(g_\theta;\ell)=\frac{1}{N}\sum_{i=1}^N \ell\big(\bar y_i,g_\theta(x_i)\big).
$$

The framework surrounding ICM is situated within an established family of CLL risk estimators and surrogate losses. Unbiased Risk Estimators (URE) were shown to exist under a uniform transition matrix, meaning complementary labels are uniformly drawn from classes other than the ordinary label, but these estimators can be unstable and can suffer from negative empirical risks. To avoid that instability, surrogate complementary loss methods optimize convex surrogates tied directly to complementary labels. Two instantiations used with ICM are the negative-learning variant
$$
\ell_{\mathrm{NL}}(\bar y,p)=-\log\big(1-p_{\bar y}+\gamma\big), \qquad p=\mathrm{softmax}(g(x)),
$$
with $\gamma\in(0,1)$, and the exponential variant
$$
\ell_{\mathrm{EXP}}(\bar y,p)=\exp(p_{\bar y}).
$$
For non-uniform complementary-label generation, Forward Correction (FWD) introduces a transition layer,
$$
\bar f(x)=T^\top f(x),\qquad \ell_{\mathrm{FWD}}(\bar y,p)=\ell\big(\bar y,T^\top p\big).
$$
The distribution-matching loss DM is also used as a representative CLL baseline [2509.17971].

This formulation is important because ICM is not a standalone classifier. It is an augmentation procedure designed to plug into loss-based CLL training without changing the underlying task definition.

## 2. Why vanilla Mixup fails in CLL

Vanilla Mixup in ordinary supervised learning forms a synthetic input-target pair from samples $(x_i,y_i)$ and $(x_j,y_j)$ using $\lambda\sim\mathrm{Beta}(\alpha,\alpha)$:
$$
\tilde x=\lambda x_i+(1-\lambda)x_j,\qquad
\tilde y=\lambda y_i+(1-\lambda)y_j.
$$
In the ordinary-label setting, the soft target is consistent with the mixed input. In CLL, however, the ordinary labels are unavailable. If one naively substitutes complementary labels,
$$
\tilde{\bar y}=\lambda \bar y_i+(1-\lambda)\bar y_j,
$$
the resulting soft target may assign positive mass to a true class of one of the mixed inputs. The paper identifies this as complementary-label noise [2509.17971].

The risk decomposition used in the paper formalizes this failure mode. Let the hard complementary-label risk and the soft risk under Mixup be
$$
\mathcal{R}_{\mathrm{hl}}(g;\ell)=\frac{1}{N}\sum_{i=1}^N \ell\big(\bar y_i,g(x_i)\big),\qquad
\mathcal{R}_{\mathrm{sl}}(g;\ell)=\frac{1}{N}\sum_{(i,j)}\ell\big(\tilde{\bar y}_{i,j},g(\tilde x_{i,j})\big).
$$
An error term due to complementary-label noise is defined as
$$
\varepsilon=\mathbb{E}_{(x,\bar y)\sim \bar D}\big[\bar y=f(x)\big].
$$
For a synthetic pair $(\tilde x_{i,j},\tilde{\bar y}_{i,j})$, the proposition in the paper shows that the mixed complementary risk includes a clean term plus $\lambda\varepsilon_i+(1-\lambda)\varepsilon_j$, which captures additional error arising when complementary labels coincide with true classes under mixing [2509.17971].

The empirical evidence reported is specific and substantial. On CIFAR-10 with ResNet18 and SCL-NL, naive Mixup induces approximately $15.8\%$ complementary-label noise, and removing that noise improves accuracy by approximately $7\%$. A gradient-estimation analysis further compares the ordinary gradient $f=\nabla \ell(y,g(x))$ with the complementary gradient $c=\nabla \bar \ell(\bar y,g(x))$ through
$$
\mathrm{MSE}=\mathbb{E}\big[(f-c)^2\big]
=\underbrace{\mathbb{E}\big[(f-b)^2\big]}_{\mathrm{Bias}^2}
+\underbrace{\mathbb{E}\big[(b-c)^2\big]}_{\mathrm{Variance}},
$$
where
$$
b=\frac{1}{K-1}\sum_{y'\ne y}\nabla \bar\ell\big(y',g(x)\big).
$$
The “Mixup Noise-Free” setting shows significantly lower MSE, bias, and variance than naive Mixup on MNIST and CIFAR-10. This suggests that the central deficiency of vanilla Mixup in CLL is not interpolation itself, but interpolation across semantically incompatible complementary labels [2509.17971].

## 3. ICM algorithm and learning objective

ICM restricts Mixup to nearby examples that are likely to share the same ordinary class. The method first extracts embeddings with a pretrained SimSiam encoder and then performs $k$-means clustering on the embedding set $\{z_i\}$. Each sample receives a cluster label $c_i\in\{1,\dots,k\}$ as a preprocessing step. Euclidean distance is used in embedding space, and the clustering objective minimizes within-cluster sum of squares [2509.17971].

Within training, each minibatch $B$ is partitioned into cluster-specific sub-batches $\{B_u\}$. For each cluster $u$ with $|B_u|\ge 2$, distinct pairs $(x_i,\bar y_i,u)$ and $(x_j,\bar y_j,u)$ are sampled. Using $\lambda\sim\mathrm{Beta}(\alpha,\alpha)$ with $\alpha>0$, ICM forms
$$
\tilde x_{i,j}=\lambda x_i+(1-\lambda)x_j,\qquad
\tilde{\bar y}_{i,j}=\lambda \bar y_i+(1-\lambda)\bar y_j.
$$
For stability, $\alpha\in[0.1,0.4]$ is reported to work well in practice, and the implementation sets $\lambda_x=\lambda_{\bar y}$ to synchronize input and label mixing.

The training objective is the augmented CLL loss
$$
\mathcal{L}'(g_\theta;\ell)
=\frac{1}{N}\sum_{(i,j)}\ell\big(\tilde{\bar y}_{i,j},g_\theta(\tilde x_{i,j})\big)
=\frac{1}{N}\sum_{(i,j)}\Big[
\lambda\,\ell\big(\bar y_i,g_\theta(\tilde x_{i,j})\big)
+(1-\lambda)\,\ell\big(\bar y_j,g_\theta(\tilde x_{i,j})\big)
\Big].
$$
In this formulation, complementary labels remain one-hot over incorrect classes before mixing; the convex interpolation is intended to keep the mixed target unlikely to allocate probability to a true class when both examples come from the same latent class cluster [2509.17971].

The significance of ICM lies in how it reframes augmentation for complementary supervision. Standard Mixup assumes label interpolation is semantically valid across random pairs. ICM replaces that assumption with a locality prior in a learned representation space, using cluster consistency as a surrogate for latent class consistency.

## 4. Integration with CLL losses and implementation details

A key characteristic of ICM is that it is designed to slot directly into existing CLL objectives. For SCL-NL and SCL-EXP, hard complementary labels are replaced with soft $\tilde{\bar y}$ and passed into the chosen loss:
$$
\ell_{\mathrm{NL}}(\tilde{\bar y},p)
=\lambda\,\ell_{\mathrm{NL}}(\bar y_i,p)+(1-\lambda)\,\ell_{\mathrm{NL}}(\bar y_j,p),
$$
$$
\ell_{\mathrm{EXP}}(\tilde{\bar y},p)
=\lambda\,\ell_{\mathrm{EXP}}(\bar y_i,p)+(1-\lambda)\,\ell_{\mathrm{EXP}}(\bar y_j,p).
$$
For FWD, the same construction is used after the transition transform:
$$
\ell_{\mathrm{FWD}}(\tilde{\bar y},p)
=\ell\big(\tilde{\bar y},T^\top p\big)
=\lambda\,\ell(\bar y_i,T^\top p)+(1-\lambda)\,\ell(\bar y_j,T^\top p).
$$
For DM, ICM provides the synthetic pairs and the loss is computed on the mixed inputs and mixed complementary targets without further changes. The paper states that no modification is needed beyond replacing random cross-class Mixup with within-cluster sampling and using soft complementary targets [2509.17971].

The implementation protocol is fixed at the dataset level. Embeddings are extracted once with a pretrained SimSiam encoder, clustering is performed once with $k$-means, and cluster IDs are stored. Typical experiments use $k=50$, selected via validation, with an ablation over $k\in[30,90]$ recommending $k=50$ as a good accuracy-efficiency trade-off. The training loop then repeatedly samples minibatches, partitions them by cluster, forms intra-cluster mixed examples, accumulates the selected CLL loss on the synthetic pairs, and updates parameters by gradient descent [2509.17971].

Recommended settings reported for reproducibility include a ResNet18 backbone, Adam, initial learning rate $1\mathrm{e}{-4}$, weight decay $1\mathrm{e}{-4}$, batch size $512$, and $300$ epochs. Dataset-specific $\alpha$ values are also reported: CIFAR-10 approximately $0.4$, CIFAR-20 approximately $0.1$, MNIST approximately $0.1$, KMNIST approximately $0.3$, and FMNIST approximately $0.1$ [2509.17971].

## 5. Empirical behavior across balanced and imbalanced CLL

The evaluation spans synthetic complementary labels on CIFAR-10, CIFAR-20, MNIST, KMNIST, and FMNIST, together with real human-annotated complementary labels on CLCIFAR10 and CLCIFAR20. Balanced and long-tailed imbalanced settings are considered, including $\rho=1$, $\rho=10$, and $\rho$ up to $100$, under three setups: imbalanced ordinary labels with uniform transition, balanced ordinary labels with biased transition, and both imbalanced ordinary labels and biased transition [2509.17971].

Representative reported results are summarized below.

| Setting | Baseline vs. +ICM | Change |
|---|---|---|
| Balanced CLCIFAR10 | FWD $42.98\pm0.36$ vs. FWD+ICM $58.97\pm0.21$ | $+16.0$ |
| Balanced CLCIFAR10 | S-NL $37.59\pm0.40$ vs. S-NL+ICM $56.63\pm0.61$ | $+19.0$ |
| Balanced CLCIFAR20 | FWD $21.10\pm0.23$ vs. FWD+ICM $35.94\pm0.33$ | $+14.8$ |
| CLCIFAR10, $\rho=10$ | S-NL $17.77\pm0.20$ vs. S-NL+ICM $28.44\pm0.05$ | $+10.7$ |
| CIFAR-10, $\rho=100$ | S-NL $22.41\pm0.31$ vs. S-NL+ICM $36.21\pm0.19$ | $+13.8$ |
| CIFAR-10, $\rho=1$ | FWD $64.29\pm0.33$ vs. FWD+ICM $79.22\pm0.03$ | $+14.9$ |
| CIFAR-20, $\rho=1$ | FWD $23.18\pm0.34$ vs. FWD+ICM $42.20\pm0.09$ | $+19.0$ |
| MNIST, $\rho=100$ | S-NL $50.15\pm0.52$ vs. S-NL+ICM $85.83\pm0.19$ | $+35.7$ |

Across methods, naive “+Mix” provides only limited benefit and is consistently worse than “+ICM”; one reported example is CIFAR-10 with $\rho=1$, where S-EXP+Mix obtains $71.72$ while S-EXP+ICM reaches $78.86$, a gain of $7.1$. The paper also reports a marked drop in complementary-label noise ratio relative to Mixup, such as MNIST decreasing from $16.24\%$ to $0.95\%$ under ICM. Combining ICM with Flip, Cutout, AutoAugment, and RandAug yields accuracy near $80\%$ on CIFAR-10, and ICM consistently boosts performance over the same augmentations without ICM [2509.17971].

These results support two linked conclusions. First, ICM is not tied to one particular CLL loss, since gains are reported for SCL-NL, SCL-EXP, FWD, and DM. Second, the effect is strongest where complementary-label noise is especially damaging, which is consistent with the paper’s broader summary that ICM improves MNIST-family accuracy by approximately $30\%$ and CIFAR-family performance by approximately $10$–$20\%$.

## 6. Computational profile, limitations, and related formulations

The computational overhead of ICM is concentrated in preprocessing. SimSiam embedding extraction requires one pass over the dataset, and $k$-means clustering has complexity $O(NkI)$ with $k$ clusters and $I$ iterations. Both are performed once offline. During training, sampling intra-cluster pairs within minibatches is reported to add negligible overhead relative to forward and backward passes. For large $N$, approximate $k$-means or mini-batch $k$-means can be used, and cluster IDs $c_i$ impose minimal memory overhead [2509.17971].

The method has several explicit limitations. If embeddings do not reflect class structure, as can occur with weak features or insufficient pretraining, clusters may overlap and intra-cluster mixing may not reduce noise. In datasets with heavy inter-class overlap, intra-cluster mixing may still induce noise. Performance depends on the choice of $k$, $\alpha$, and embedding quality; too small a $k$ yields coarse clusters, while too large a $k$ produces tiny clusters. ICM also creates more complex synthetic structures, and simpler models such as linear models or small MLPs can underperform with ICM on simple datasets such as the MNIST family. The multi-complementary-label setting was not evaluated and is left as future work [2509.17971].

For severe imbalance, the paper reports an extension called MICM that mixes three samples inside clusters using $\mathrm{Dirichlet}(\alpha,\alpha,\alpha)$ for inputs and Inverse Distance Weighting for label coefficients:
$$
\tilde x_{i,j,k}=\lambda_1 x_i+\lambda_2 x_j+\lambda_3 x_k,
$$
$$
\tilde{\bar y}_{i,j,k}
=\lambda_{\bar y_1}\bar y_i+\lambda_{\bar y_2}\bar y_j+\lambda_{\bar y_3}\bar y_k.
$$
The weighting uses
$$
\lambda_{\bar y_i}
=\frac{1/d(\tilde x,x_i)}{\sum_{\ell\in\{i,j,k\}}1/d(\tilde x,x_\ell)},
$$
with $C\approx 30$ used when $\|\tilde x-x_i\|_2=0$ to avoid division by zero and control anchor weight. MICM is reported to improve minority-class learning on complex datasets such as CIFAR, while ICM suffices on simpler datasets such as the MNIST family [2509.17971].

A related but distinct formulation appears in "SynerMix: Synergistic Mixup Solution for Enhanced Intra-Class Cohesion and Inter-Class Separability in Image Classification", where SynerMix-Intra is described as “ICM at the class granularity” in an ordinary supervised setting. There, features from the same class are combined with normalized Uniform$(0,1)$ weights, labels remain hard one-hot targets, and the method is combined with inter-class MixUp or Manifold MixUp through a weighted objective. That paper states that “Intra-Cluster Mixup (ICM) is intra-class mixing when the ‘cluster’ is the entire class,” and suggests adaptation to true sub-cluster ICM by replacing whole-class sets with sub-clusters defined within a class [2403.14137]. This suggests that the term “intra-cluster mixup” has a broader geometric interpretation beyond CLL: local convex combinations are used to promote label-consistent interpolation, but the label semantics differ substantially between ordinary supervision and complementary-label supervision.

Within the CLL literature, ICM’s novelty is therefore not merely that it performs local Mixup, but that it operationalizes locality specifically to respect the complementary-label constraint. That places it alongside URE, SCL, FWD, and DM as a method for complementary supervision, while addressing a part of the training pipeline—data augmentation—that earlier CLL work had largely left underexplored [2509.17971].

Source: https://www.emergentmind.com/topics/intra-cluster-mixup-icm