Contrastive Clustering Methods

• Contrastive clustering is a technique that jointly learns feature embeddings and cluster assignments through contrastive objectives, capturing both local and global data structures.
• It employs multi-level contrast at instance and prototype levels using data augmentations and various backbones like CNNs and transformers to form semantic groupings.
• Practical implementations demonstrate state-of-the-art performance in image, text, and graph clustering, offering scalability and enhanced interpretability.

Contrastive clustering is a family of unsupervised learning techniques in which representation learning via contrastive objectives is directly integrated with the formation of clusters, enabling the discovery of meaningful semantic groupings in complex data modalities including images, text, graphs, and time series. The core innovation is the simultaneous exploitation of local (instance-level) and global (cluster-level or prototype-level) structure within the data, leveraging powerful neural backbones and recently, transformer variants, to learn highly discriminative feature spaces suitable for clustering without labels.

1. Core Principles of Contrastive Clustering

Contrastive clustering formalizes clustering as a problem of jointly learning feature embeddings and cluster assignments by optimizing contrastive losses that operate at several structural levels of the data. The predominant architecture involves a backbone encoder (CNN, RNN, or ViT), an instance-level projector for InfoNCE-style learning, and a cluster-level or prototype projector for enforcing global structure. Stochastic data augmentations or learnable Siamese pathways are used to create multiple views of each object, establishing positive and negative pairs for the contrastive procedure.

At the instance level, the network is trained to maximize the similarity between different augmentations of the same sample (positive pairs) and minimize similarity with other samples (negatives). At the cluster level, either assignment vectors (soft or hard) or cluster prototype representations are enforced to be consistent across different views, often augmented by entropy regularizers to prevent degenerate solutions. Several works explicitly interpret the assignment matrix as a soft labeling of instances (rows) and the cluster prototype matrix as a set of global descriptors (columns), enforcing consistency in both spaces (Li et al., 2020).

The loss landscape often combines InfoNCE terms over instance pairs, InfoNCE or MSE terms over prototype/cluster assignments, and regularization promoting full cluster utilization. Examples include the CC/VTCC/SACC frameworks for images (Li et al., 2020, Ling et al., 2022, Deng et al., 2022), SCCL for short text (Zhang et al., 2021), DTCC for time series (Zhong et al., 2022), and CPCC for prototype-centric clustering (Dong et al., 21 Aug 2025).

2. Methodological Innovations and Variants

Contrastive clustering has diversified rapidly along architectural, augmentation, and loss design axes.

• Multi-Level Contrast: Most modern methods employ both instance-level and cluster/prototype-level contrastive terms. For example, SACC utilizes a three-view scheme (two weak, one strong augmentation) to enhance invariance and semantic richness (Deng et al., 2022). Cluster assignments are regularized using entropy penalties and cross-view consistency.
• Prototype/Soft Prototype Contrasts: Methods such as CPCC employ soft prototype aggregation, where prototypes are weighted means (often quadratically-weighted) of feature vectors according to their assignment confidence, reducing prototype drift and inter-class conflict (Dong et al., 21 Aug 2025). Contrastive loss is then formulated at the prototype level rather than the instance level.
• Momentum and Teacher Networks: MCC and its federated variant employ BYOL-style momentum-updated target networks to stabilize contrastive clustering, managing distributed and non-IID data (Miao et al., 2022).
• Graph Domain Extensions: Recent schemes such as SCGC, CCGC, and Congregate adapt the contrastive clustering paradigm to non-Euclidean data. These employ Siamese or augmentation-free encoders with various positive/negative sampling strategies guided by intrinsic structure (e.g., high-confidence clusters or Ricci curvature), and may operate in specialized geometric spaces (e.g., product manifolds of constant curvature) (Yang et al., 2023, Liu et al., 2022, Sun et al., 2023).
• Transformer-based Advances: VTCC and MFAVBs-CC introduce ViT backbones, integrating convolutional stems to enhance patch stability and multi-stage fusion blocks to explicitly exploit complementary information between data views (Ling et al., 2022, Wang et al., 12 Nov 2025).
• Semi-Supervised and Personalized Clustering: OCC incorporates oracle-guided pairwise supervision into the contrastive loss, enabling orientation-aware (personalized) clustering via informative active queries (Wang et al., 2022).
• Multi-View and Cross-Modal Extensions: DWCL introduces best-view and dual-weighting strategies for contrastive learning across multiple data views, mitigating the degeneration and unreliability issues that plague naïve cross-view aggregation (Yuan et al., 26 Nov 2024).

Summary Table: Key Components Across Exemplars

Method	Backbone	Positive Pairing	Cluster Loss	Notable Innovations
CC, SACC, VTCC	CNN, ViT	Augmentations	InfoNCE+Entropy	Dual-level loss (instance & cluster)
CPCC	CNN	Soft prototype weights	Prototype-level contrast	Quadratically-weighted prototypes
MCC, FedMCC	CNN (BYOL style)	Momentum twin networks	Instance & cluster	Federated aggregation, stability
SCGC, CCGC	MLPs (graphs)	Siamese, high-conf clusters	Cross-view MSE/contrastive	Graph-specific pair selection
CongreGate	fRGCN (manifolds)	Geometric views	Ricci, reweighted contrast	Heterogeneous curvature spaces
OCC	CNN	Oracle-augmented pairs	Active contrastive loss	Personalized clustering, risk bounds
DWCL	Any, Multi-view	Best-view B-O selection	Dual-weighted InfoNCE	View quality/discrepancy weighting

3. Loss Formulations and Theoretical Foundations

Contrastive clustering loss functions most often generalize the InfoNCE objective to include not only instance-wise but also cluster/prototype-level information, with regularization to encourage balanced usage of clusters and avoidance of collapse. For example, the CC framework optimizes

$L_{\text{total}} = L_{\text{ins}} + L_{\text{clu}}$

where $L_{\text{ins}}$ is a symmetric instance-level InfoNCE loss, and $L_{\text{clu}}$ is a prototype-level InfoNCE term with additional entropy maximization (Li et al., 2020). SACC generalizes this to multiple views and combines instance and cluster terms over all view pairs, while CPCC replaces instance-level contrast with prototype-level contrast using quadratically-weighted soft assignments (Dong et al., 21 Aug 2025).

In the theoretical domain, recent work explicitly connects standard contrastive learning (SimCLR, InfoNCE) to spectral clustering on the similarity/augmentation graph. The loss

$$L_{\text{NCE}}(Z) = - \sum_{i,j} \pi_{ij} \log \frac{K_{Z,ij}}{\sum_k K_{Z,ik}}$$

can be shown, up to a mild repulsion term, to be equivalent to minimizing $\mathrm{Tr}(Z^\top L(\Pi) Z)$, i.e., the Laplacian quadratic form of the data augmentation graph $\Pi$, which underpins spectral clustering (Tan et al., 2023). This perspective yields practical implications for kernel design (e.g., mixtures of Gaussian and Laplacian kernels) and for understanding clustering collapse.

For multi-view scenarios, DWCL provides a mathematical framework in which weighting the cross-view InfoNCE loss by per-view silhouette (quality) and cluster assignment similarity (CMI) weights provably tightens the mutual information bound and reduces computational complexity from $O(V^2)$ to $O(V)$, while robustly filtering out weak or inconsistent views (Yuan et al., 26 Nov 2024).

4. Practical Architectures and Optimization

Contrastive clustering architectures are now standardized around efficient discriminative backbones (ResNet-34, ViT-small, dilated RNNs for sequence), projectors for both instance and cluster/prototype levels, and batch-wise optimizations with in-batch negatives. Representative augmentations cover both weak (crop, jitter, flip) and strong (AutoContrast, Solarize, etc.) families; multi-view and cross-modal strategies rely on either independent encoders or shared-weight pathways.

Training is typically performed using Adam or SGD with mild learning rate schedules, standard mini-batch sizes (128–400), and extended epoch counts (500–1000). For CPCC and MCC/FedMCC, momentum update and early stage pretraining on stability-inducing objectives are critical for convergence and avoidance of trivial solutions. Graph-based methods such as SCGC decouple costly convolutional steps into preprocessing, permitting dramatic speedups (up to 7× faster than GCN-based methods) (Liu et al., 2022).

Prototype-based methods frequently require clustering assignments to compute centroids, often via K-means or explicit soft assignment kernels (e.g., Student's t-distribution). Entropy and cluster balancing penalties are ubiquitous to enforce uniform cluster utilization and prevent collapse.

5. Empirical Performance and Comparative Results

Contrastive clustering techniques consistently outperform classic deep clustering and conventional self-supervised representation learning pipelines across a wide array of benchmarks. For example:

• In text, SCCL achieves improvements of +3–11% in ACC and +4–15% NMI over strong baselines on short text clustering tasks (Zhang et al., 2021).
• In image clustering, SACC (with both strong and weak augmentations) surpasses previous state-of-the-art performance on benchmarks such as CIFAR-10/100, STL-10, ImageNet-10, and ImageNet-Dogs, obtaining up to 76.5% NMI on CIFAR-10 and 87.7% on ImageNet-10 (Deng et al., 2022).
• Prototype-centric CPCC yields further improvements, achieving 0.950 ACC on CIFAR-10 and 0.962 on ImageNet-10, consistently beating prior ProPos and PCL methods while showing robustness to noisy assignments and parameter settings (Dong et al., 21 Aug 2025).
• On graph domains, SCGC and CCGC demonstrate state-of-the-art accuracy and NMI across all major citation and Wikipedia benchmarks, with improved speed and discrimination owing to structure-aware pair construction and high-confidence positive selection (Liu et al., 2022, Yang et al., 2023).
• Federated and decentralized scenarios are addressed by FedMCC, which shows up to +11.8 points ACC improvement over best prior federated clustering methods on CIFAR-10 (Miao et al., 2022).
• Vision Transformer-based VTCC and MFAVBs-CC architectures achieve superior performance on challenging natural and remote-sensing datasets, with MFAVBs-CC attaining an average gain of +0.087 ACC and +0.072 NMI over VTCC across seven datasets (Wang et al., 12 Nov 2025).

6. Theoretical Insights, Limitations, and Extensions

The formal equivalence between contrastive objectives (InfoNCE) and spectral clustering frames the learning of discriminative clustering-friendly embeddings as Laplacian minimization tasks on augmentation or affinity graphs (Tan et al., 2023). This theory directly informs kernel selection and diagnosis of failure modes such as cluster collapse, suggesting that control of pairwise similarity and augmentation graphs is paramount.

Major practical limitations include sensitivity to augmentation policy (particularly in text and graphs), sensitivity to the selection of cluster count $K$, and, for some methods, computational bottlenecks in assignment or prototype updates. Prototype-based clustering introduces risk of prototype drift, mitigated in CPCC by confidence-weighted aggregation. Oracle-guided or semi-supervised scenarios demand efficient active querying mechanisms and robust theoretical risk bounds (Wang et al., 2022).

Recent innovations such as dual consistency learning, federated aggregation, augmentation-free geometric contrast (CongreGate), and dynamic per-view weighting open further research directions in robustification, multimodality, and scaling to large non-IID datasets.

7. Outlook and Open Problems

Contrastive clustering has established itself as the leading method for unsupervised clustering in high-dimensional, complex domains. Active research continues in the following directions:

• Extension to heterogeneous, multimodal, and multi-view data with robust, theoretically-justified weighting and alignment schemes (Yuan et al., 26 Nov 2024).
• Adaptive, online determination of the number of clusters and dynamic regularization.
• Better integration of geometric and structural priors, such as Riemannian-curvature embeddings for graphs (Sun et al., 2023).
• Handling of domain-mismatched, extremely fine-grained, or non-stationary data.
• Stronger theoretical foundations for prototype and assignment stability.
• Efficient large-scale deployment, particularly federated and decentralized clustering under strict communication and privacy constraints.

Contrastive clustering continues to bridge the gap between representation discrimination and cluster assignment, with state-of-the-art performance in challenging domains and a robust, theoretically grounded formulation drawing upon decades of clustering, spectral graph, and metric learning theory.