Contrastive Clustering Overview

• Contrastive Clustering (CC) is an unsupervised deep learning paradigm that unifies instance-level discrimination and semantically coherent cluster formation.
• It leverages dual objectives—instance-level contrastive loss and cluster-level contrastive loss—ensuring both local similarity and global semantic grouping.
• Recent extensions, including prototype learning and graph-based techniques, have achieved state-of-the-art results in image, graph, and speech domains.

Contrastive Clustering (CC) is an unsupervised deep learning paradigm that unifies contrastive learning and clustering by jointly optimizing for both instance-level discrimination and semantically coherent cluster formation. CC leverages the strengths of contrastive objectives—pulling together representations of positive pairs and separating negatives—while integrating explicit mechanisms for discovering clustering structure. By coupling instance- and cluster-level contrastive signals with prototype learning, entropy regularization, and (in recent variants) sophisticated sampling or graph-theoretic mechanisms, CC has set the state of the art across image, graph, speech, and fine-grained domains.

1. Fundamental Principles and Model Structures

Contrastive Clustering starts from the standard contrastive learning protocol: two stochastic augmentations are applied to each instance, forming a positive pair, while all other augmented samples in the batch are treated as negatives. A backbone encoder maps each augmentation to a feature representation, which is then processed by separate instance-level and cluster-level projection heads. The instance head maps features to a normalized latent space for instance-wise InfoNCE loss, while the cluster head predicts soft assignments over $M$ clusters (via a softmax output) and supports a second "cluster-level" contrastive loss that aligns cluster prototypes across views (Li et al., 2020).

Formally, for a batch of $N$ samples $X = \{x_i\}_{i=1}^N$, each with two augmentations $(x_i^a, x_i^b)$, the model computes:

• $z_i^a = g_I(f(x_i^a))$ (instance-level embedding)
• $y_i^a = g_C(f(x_i^a))$ (cluster-level soft assignment)

where $f$ is the shared encoder, $g_I$ the instance projector, and $g_C$ the cluster head.

This dual mechanism enables end-to-end training: the instance-level loss ensures local structural discrimination, while the cluster-level loss encourages aggregation into semantic groups. The resultant feature matrix $Y$ admits a dual interpretation:

• Rows $y_i$ as soft cluster labels for instances,
• Columns $y_{:j}$ as cluster prototypes.

This perspective underpins vertically and horizontally structured contrastive objectives that establish both sharp intra-cluster compactness and global feature separation (Li et al., 2020, Sadeghi et al., 2022).

2. Core Objectives and Prototype Contrast

The primary losses in CC are:

1. Instance-level contrastive loss (InfoNCE/NT-Xent):

$$\mathcal{L}_{\text{ins}} = \frac{1}{2N} \sum_{i=1}^N \Big\{ -\log \frac{\exp(s(z_i^a,z_i^b)/\tau_I)}{\sum_{j=1}^N [\exp(s(z_i^a,z_j^a)/\tau_I) + \exp(s(z_i^a,z_j^b)/\tau_I)]} + (a \leftrightarrow b) \Big\}$$

enforcing that augmented versions of the same instance are close, while others are negative samples.

2. Cluster-level contrastive loss:

$$\mathcal{L}_{\text{clu}} = \frac{1}{2M}\sum_{j=1}^M \Big\{ -\log \frac{\exp(s(y_j^a, y_j^b)/\tau_C)}{\sum_{l=1}^M [\exp(s(y_j^a, y_l^a)/\tau_C)+\exp(s(y_j^a, y_l^b)/\tau_C)]} + (a \leftrightarrow b) \Big\} - H(Y)$$

where $H(Y)$ is an entropy term for regularizing the marginal cluster assignments and avoiding collapse.

3. Combined objective:

$$\mathcal{L}_{\text{CC}} = \mathcal{L}_{\text{ins}} + \mathcal{L}_{\text{clu}}$$

Recent methodological advances, such as Center-Oriented Prototype Contrastive Clustering (CPCC), replace hard prototype averages with soft, confidence-weighted prototypes:

$$w_{ik} = \frac{q_{ik}^2 / f_k}{\sum_{k'} q_{ik'}^2 / f_{k'}}$$

prototypes are then computed based on these weights, sharply reducing drift and false negative effects (Dong et al., 21 Aug 2025). The prototype-prototype contrastive objective

$$L_{\text{proto}} = -\frac{1}{K}\sum_k \log \frac{\exp(\langle p_k, p_k' \rangle / \tau)}{\sum_{j \neq k} [\exp( \langle p_k, p_j \rangle / \tau ) + \exp( \langle p_k, p_j' \rangle / \tau ) ]}$$

enables high-confidence samples to anchor cluster representations more robustly than uniform prototypes.

3. Extensions: Cross-Instance Mining, Graphs, and Curriculum

Contrastive Clustering's reach has extended beyond IID data, with sophisticated strategies addressing both data and algorithmic limitations:

• Cross-instance positive mining (C3): Once an initial embedding is learned, cosine similarity in the latent space is used to define additional positive pairs above a threshold $\zeta$, and a weighted scheme for negatives emphasizing boundary samples. This reduces the false negative rate and improves intra-cluster compactness (Sadeghi et al., 2022).
• Graph Contrastive Clustering: Multiple graph-specific CC variants exist. SCGC uses low-pass denoising and two unshared MLPs, with a cross-view MSE loss matching structural proximity in the adjacency matrix (neighbors to 1, non-neighbors to 0); CCGC samples positives/negatives based on high-confidence clusters, and THESAURUS integrates fused Gromov–Wasserstein OT with learnable semantic prototypes, directly aligning the clustering pretext task and mitigating assimilation/uniformity in rare classes (Liu et al., 2022, Yang et al., 2023, Deng et al., 2024).
• Multi-task curriculum and entropy guidance: CCGL dynamically transitions nodes from discrimination to clustering, guided by per-node (soft) assignment entropy, and adjust graph augmentations to preserve class structure based on low-entropy pseudo-labels. This "curriculum" approach yields flexible sample selection and robust generalization (Zeng et al., 2024).
• Counterfactual hard negative generation (MeCoLe): Decoupling class-dependent/invariant node features, CC is applied to competitive negatives synthesized by altering class-dependent features, focusing learning on near-boundary samples for greater cluster separation (Cui et al., 2024).
• Speech/data modalities and batch-wise adaptive clustering: In CCC-wav2vec 2.0, batch-level mini-k-means identifies easy negatives ("same cluster" as positive) and down-weights their contribution, while cross-contrastive configurations further filter the learning signal to sharpen speech segment distinctions (Lodagala et al., 2022).

4. Hierarchical and Prototype Mechanisms

Recent work moves beyond flat cluster assignments:

• Contrastive Hierarchical Clustering (CoHiClust): Builds a soft binary tree where each route corresponds to a latent cluster, composing instance similarities at multiple levels and combining NT-Xent regularization with tree-wise contrastive objectives. The learned tree exposes cluster relationships and supports multi-resolution partitioning and dendrogram purity evaluation (Znaleźniak et al., 2023).
• Prototype learning and explicit cluster centroids: Several frameworks (e.g., cluster-center predictor in (Sundareswaran et al., 2021), soft prototype contrast in (Dong et al., 21 Aug 2025)) use learnable or adaptive centroids, with Student’s-t or soft assignment, and sharpened target distributions to encourage geometric separation of cluster embeddings.

5. Performance and Empirical Outcomes

Contrastive Clustering and its variants have established state-of-the-art metrics across a range of visual, graph, and speech benchmarks:

Method	Dataset (CIFAR-10)	NMI	ACC	ARI
Best Baseline (pre-CC)		0.591	0.696	0.512
CC (Li et al., 2020)		0.705	0.790	0.637
C3 (Sadeghi et al., 2022)		0.743	0.836	0.703
CPCC (Dong et al., 21 Aug 2025)		0.900	0.950	0.898

On graph benchmarks, methods like THESAURUS, SCGC, and CCGL outperform generative, adversarial, and earlier contrastive approaches across CORA, CITESEER, AMAP, and AIR-TRAFFIC datasets (Deng et al., 2024, Liu et al., 2022, Zeng et al., 2024).

Ablation studies consistently show that removing cluster-level contrast, prototype weighting, or cross-instance mining leads to marked degradation in cluster performance, confirming the necessity of each component for robust CC.

6. Limitations, Open Problems, and Future Directions

Limitations identified across the literature include:

• Fixed cluster number $K$ is typically required; dynamic or nonparametric estimation via Dirichlet processes, adaptive prototypes, or hierarchical extensions remains open (Dong et al., 21 Aug 2025, Znaleźniak et al., 2023).
• Periodic (mini-)batch k-means or prototype updating can be computationally intensive for very large datasets; online prototype maintenance or moving-average cluster centers are active areas for improvement (Dong et al., 21 Aug 2025).
• Many methods assume reasonably informative initial representation space; CC with poor warm-up or extreme class imbalance may surface confirmation biases or collapse (Sadeghi et al., 2022, Yang et al., 2023).
• For graph CC, most approaches to date are limited to static graphs with fixed attributes; extensions to dynamic, attributed, and heterogeneous graphs are needed (Liu et al., 2022, Yang et al., 2023, Deng et al., 2024).
• Real-world deployment may require self-tuning of augmentation strength, temperature, threshold parameters, and curriculum pace for optimal robustness.

Future directions include:

• Incorporation of moving-average or online prototype learning (Dong et al., 21 Aug 2025).
• Automatic cluster enumeration and nonparametric objectives (Znaleźniak et al., 2023, Deng et al., 2024).
• Multimodal, cross-domain, or cross-modal contrastive clustering (image-text, audio-visual).
• Generalization to self-supervised settings on massive or non-IID datasets (e.g., federated/few-shot environments) (Miao et al., 2022).
• Deeper integration of robust, structure-aware negative mining, curriculum adaptation, and domain-specific augmentations.

7. Representative Implementations and Codebases

Code for several representative CC frameworks is available:

• CPCC: https://github.com/LouisDong95/CPCC (Dong et al., 21 Aug 2025)
• CCC-wav2vec 2.0: https://github.com/your-code/ (as stated in (Lodagala et al., 2022))
• CCGC and SCGC: as referenced in their respective papers (Yang et al., 2023, Liu et al., 2022)

Many of these repositories integrate both instance- and cluster-level components, data/graph augmentation pipelines, and evaluation scripts for standard clustering metrics (NMI, ACC, ARI, F1).

---

Contrastive Clustering has established a principled, extensible foundation for unsupervised structure discovery in high-dimensional data, and ongoing research continues to address its limitations and enlarge its scope across modalities and datatypes (Li et al., 2020, Dong et al., 21 Aug 2025, Sadeghi et al., 2022, Deng et al., 2024).