Mean-Shifted Contrastive Loss

• Mean-Shifted Contrastive Loss is a modification of contrastive loss that recenter features by subtracting the empirical mean or aggregating local neighborhoods to better preserve cluster structure.
• It mitigates challenges like feature collapse and over-uniformization, enabling robust fine-tuning of pre-trained representations in tasks such as anomaly detection and category discovery.
• Empirical results show MSC achieves state-of-the-art performance on benchmarks like CIFAR-10 and DIOR, improving both clustering accuracy and discrimination in deep learning models.

Mean-Shifted Contrastive Loss (MSC) is a modification of the traditional contrastive loss paradigm, designed to address critical challenges that arise when applying contrastive representation learning to pre-trained features—specifically in anomaly detection and clustering contexts. Classical contrastive objectives, such as those used in SimCLR, struggle to effectively fine-tune features initialized from large-scale pre-training, often resulting in poor conditioning, feature collapse, or over-uniformity. Mean-shifted contrastive frameworks recenter the feature space or adapt local feature neighborhoods, enabling robust clustering and invariance properties while preserving or enhancing discriminability for downstream tasks. This entry provides a systematic exposition of MSC, focusing on its mathematical formulation, rationale, algorithmic mechanics, empirical behavior, and broader significance in deep representation learning.

1. Mathematical Formulation and Derivation

In the conventional contrastive learning setup, let $f(x)$ denote a feature extractor (e.g., ResNet or ViT backbone), typically pre-trained and then $\ell_2$-normalized to produce feature vectors on the unit sphere. For each sample $x_i$, two augmented views $x_i$, $x_{i+B}$ are produced, and their embeddings are encouraged to be similar, while being dissimilar to other samples in the batch:

$$L_{\text{con}}(x_i, x_{i+B}) = -\log \left[ \frac{ \exp( \operatorname{sim}(f(x_i), f(x_{i+B}))/\tau ) }{ \sum_{m=1}^{2B} 1_{m \neq i}\cdot \exp( \operatorname{sim}(f(x_i), f(x_m))/\tau ) } \right]$$

where $\operatorname{sim}(u,v) = u \cdot v$ is cosine similarity and $\tau > 0$ is a temperature parameter.

Mean-Shifted Contrastive Loss applies a crucial modification. For the anomaly detection context (Reiss et al., 2021), denote the empirical mean of normalized representations over the normal training set by

$$c = \mathbb{E}_{x \sim \mathcal{X}_{\text{train}}}[f_0(x)]$$

where $f_0(x)$ is the pre-trained, $\ell_2$-normalized embedding. The mean-shift operation is simply: for each view $u_i = f(x_i) - c$, $v_i = f(x_{i+B}) - c$.

The loss is then applied in this mean-centered space:

$$L_{\text{msc}}(x_i, x_{i+B}) = -\log \left[ \frac{ \exp( \operatorname{sim}(u_i, v_i)/\tau ) }{ \sum_{m=1}^{2B} 1_{m \neq i}\cdot \exp( \operatorname{sim}(u_i, u_m)/\tau ) } \right]$$

In more generalized category discovery settings (Choi et al., 15 Apr 2024), embeddings are mean-shifted by aggregating $k$-nearest neighbors, using a weighted kernel $\phi$ over the local neighborhood:

$$z_i = \operatorname{normalize}\left( \sum_{j \in N(i)} \phi(v_j - v_i)\, v_j \right )$$

with $\phi(v_j - v_i) = 1-\alpha$ for $j=i$, $\alpha/k$ otherwise ($\alpha \in [0,1]$), and $N(i)$ the set consisting of $i$ and its $k$-nearest neighbors in embedding space.

The shifted vectors $z_i$ and $z_i^+$ (corresponding to augmented views) are then used in a SimCLR-type loss, constructing the main contrastive mean-shift objective.

2. Motivation and Theoretical Intuition

Pre-trained deep representations exhibit highly non-uniform distributions: normal data points are typically clustered together in a localized region of the unit sphere. Applying the classical contrastive objective directly in this setting leads to the following failure modes (Reiss et al., 2021):

• Aggressive Over-uniformization: Standard contrastive loss pushes all points to utilize the entire unit sphere, disrupting tight clusters and ultimately spreading even normal points apart.
• Feature Collapse: Conversely, optimization can rapidly deteriorate, causing all feature vectors to collapse to a single point.

Mean-shifting remedies these pathologies by recentering the contrastive objective:

• Local Angular Conditioning: Centering by $c$ ensures optimization focuses on angular separation relative to the dataset's empirical mode, preventing undue spreading and enforcing augmentation invariance locally.
• Preserved Cluster Integrity: Pushing negatives apart no longer increases their Euclidean distance from the cluster mean; thus, tightly packed clusters of normal data are maintained, critical for one-class classification.
• Neighborhood Aggregation (Choi et al., 15 Apr 2024): Iteratively aggregating local neighbors via mean-shift enables direct shaping of the embedding space to form more coherent, well-separated clusters, improving downstream clustering and category discovery.

3. Algorithmic Implementation

Both anomaly detection and category discovery applications employ two-phase protocols: mean-shifting embeddings and then applying contrastive losses in the shifted (centered or neighborhood-aggregated) space. Salient recipe parameters and steps are presented below.

Anomaly Detection (OCC setting, (Reiss et al., 2021))

• Backbone: Pre-trained on ImageNet (e.g., ResNet152); only selective fine-tuning (e.g., last two blocks).
• Preprocessing: $\ell_2$ normalization; compute mean vector $c$ on normal training images.
• Data Augmentation: SimCLR augmentations (random crop, color jitter, grayscale, Gaussian blur, horizontal flip).
• Optimization: SGD (weight decay $5 \times 10^{-5}$, batch size 64, $\tau = 0.25$), trained for 20–25 epochs.
• Anomaly Scoring: At test, compute $f(x)$ for test image $x$, score by $K=2$ nearest-neighbor similarity to normal training set.

Category Discovery (CMS, (Choi et al., 15 Apr 2024))

• Encoder: DINO ViT-B/16 (frozen except last block + 3-layer projection head, dim 2048).
• Mean-shift kernel: $k=8$ neighbors, $\alpha=0.5$ provides a robust neighborhood weight.
• Objective: Total loss $$\mathcal{L} = \lambda L_\text{SC} + (1-\lambda)L_\text{CMS}$$, combining a supervised contrastive loss for labeled data with the contrastive mean-shift loss for all embeddings.
• Optimization: SGD, learning rate $0.01$–$0.05$, batch size 128, weight decay $5 \times 10^{-5}$, $\tau_u=0.3$ (coarse) or $0.25$ (fine); precompute embedding datastore at each epoch.
• Clustering: Additional mean-shift iterations at inference, followed by agglomerative clustering.

4. Empirical Performance, Ablations, and Benchmarks

Extensive benchmarking demonstrates that Mean-Shifted Contrastive Loss delivers superior or state-of-the-art performance in both anomaly detection and clustering settings.

Quantitative Results on Standard Anomaly Detection Benchmarks (Reiss et al., 2021)

Model	CIFAR-10 ROC-AUC (%)	CIFAR-100 Coarse (%)	CatsVsDogs (%)	DIOR (%)	ViT backbone (CIFAR-10)
DeepSVDD	64.8	–	–	–	–
MRot	90.1	–	–	–	–
DROC	92.5	–	–	–	–
CSI	94.3	–	–	–	–
PANDA	96.2	94.1	97.3	94.3	–
MSC	97.2	96.4	99.3	97.7	98.6

Additional ablations show:

• Robustness to temperature choice: $L_{\text{msc}}$ is stable for $\tau \in [0.1, 0.5]$.
• Superior resilience to catastrophic collapse compared to classical contrastive and alternative center losses.
• Consistent gains on both single-class and multi-modal normality regimes, and across diverse architectures (ResNet18, EfficientNet, DenseNet, ViT).

Clustering and Category Discovery (Choi et al., 15 Apr 2024)

• CMS gives 10–15% absolute clustering accuracy improvement versus vanilla DINO or semi-supervised contrastive on six GCD benchmarks.
• Multiple mean-shift steps at inference further enhance clustering, yielding up to 5% additional gain.
• Ablations confirm stability for $k=4$–$16$ neighbors and kernel weight $\alpha=0.5$–$0.8$.

5. Connections to Related Methods and Theoretical Insights

MSC stands at the intersection of contrastive representation learning, kernel mean-shift mode-seeking, and self-supervised fine-tuning. Key connections and findings include:

• Center Losses: Angular center loss outperforms Euclidean center loss, reinforcing that angular deviations from the empirical mode $c$ provide sharper discriminativeness for one-class classification (Reiss et al., 2021).
• Neighborhood Aggregation: Using $k$NN-based kernels for mean-shift is empirically superior to radius- or Gaussian-based kernels for clustering stability (Choi et al., 15 Apr 2024).
• Feature Collapse Mitigation: Centering and recentering interventions are critical; naïve early stopping or EWC regularization only delay collapse under conventional contrastive training.
• Clustering Geometry: Mean-shifted contrastive training shapes the embedding space to exhibit clear basin-of-attraction behavior under further mean-shift updates, enabling automated clustering without knowing $K$ a priori.

6. Broader Impact and Applications

Mean-Shifted Contrastive Loss offers an adaptable, architecture-agnostic framework for tasks requiring either tight clustering of normal samples (as in anomaly detection) or well-separated, unsupervised category formation (as in generalized category discovery):

• Strong One-Class Classification (OCC): Nearly perfect outlier detection on datasets such as CIFAR-10, CatsVsDogs, DIOR, and MVTec.
• Generalized Category Discovery: State-of-the-art unsupervised clustering without requiring prior knowledge of the number of classes.
• Scalable with Modern Backbones: Compatible with ResNet, EfficientNet, DenseNet, ViT, and frozen DINO initializations.

The introduction of mean-shift operations—either by recentering with respect to dataset mean or aggregating local neighborhoods—enables contrastive objectives to adapt to the statistical structure and geometry of real-world feature distributions, mitigating classical failure modes and unlocking robust fine-tuning and cluster formation in deep representation learning (Reiss et al., 2021, Choi et al., 15 Apr 2024).