DirectCLR: Direct Contrastive SSL

• DirectCLR is a contrastive self-supervised learning method that prevents dimensional collapse by applying the InfoNCE loss directly on a fixed subvector of the encoder’s output.
• It eliminates the need for a trainable projector by leveraging theoretical insights into singular value dynamics and fixed low-rank representations.
• Empirical evaluations on ImageNet demonstrate that DirectCLR achieves competitive accuracy, recovering most performance of multi-layer projectors with tuned dimensionality.

DirectCLR is a contrastive self-supervised learning (SSL) method designed to address the phenomenon of dimensional collapse in joint-embedding frameworks. Unlike standard approaches such as SimCLR, which utilize an explicit trainable projector head, DirectCLR directly optimizes a subset of the representation space by applying the InfoNCE loss to a fixed subvector of the encoder’s output. This method is theoretically and empirically motivated by analysis identifying two primary sources of dimensional collapse in contrastive SSL and provides a lightweight, competitive alternative to projected representations (Jing et al., 2021).

1. Theoretical Foundations: Dimensional Collapse in Contrastive SSL

In joint embedding contrastive approaches, negative sampling averts trivial solution collapse (all embeddings identical). However, both theoretical modeling and empirical findings demonstrate that contrastive methods (such as SimCLR) still admit dimensional collapse: the learned representation occupies a lower-dimensional subspace of $\mathbb{R}^d$ despite full-capacity models and training, with many singular values of the representation covariance matrix converging to zero after training.

Two core mechanisms for dimensional collapse are established:

a. Strong Augmentation:

Under a linear network $z = Wx$ trained with gradient flow and no regularization, the update dynamics are governed by

$$\dot W = -W X \qquad X = \hat\Sigma_0 - \hat\Sigma_1,$$

where $\hat\Sigma_0$ is the empirical covariance from positive/negative data pairs, and $\hat\Sigma_1$ comes from augmented views (see Lemma 2). If augmentation drives negative eigenvalues in $X$, then analytically,

$$W(t) = W(0) \exp(Xt) \xrightarrow[t\to\infty]{} [\text{rank-deficient}],$$

causing the learned features $z$ to span a low-dimensional subspace (Theorem 3).

b. Implicit Regularization in Deep Linear Networks:

For a two-layer linear MLP $(z = W_2 W_1 x)$, even with $X \succ 0$, singular value dynamics and gradient-induced alignment (Theorems 5–6) bias the network toward aligning adjacent layer singular vectors and growing the largest singular values fastest, leaving small singular components stagnant. The emergent effect is that $z = Wx$0 becomes effectively low-rank, again yielding dimensional collapse.

2. DirectCLR Objective: Derivation and Contrasts

DirectCLR forsakes the use of a learned projector (as conventionally employed in SimCLR—the two-layer MLP or linear head after the encoder output). Instead, it computes the InfoNCE loss directly on a designated, fixed subvector of the encoder output:

Given $z = Wx$1 from a ResNet50 backbone and a hyperparameter $z = Wx$2,

$z = Wx$3

the DirectCLR loss is

$z = Wx$4

In contrast, SimCLR applies a trainable projector $z = Wx$5 (single or multilayer network), and the InfoNCE loss is situated on $z = Wx$6. DirectCLR’s key insight is that the ResNet residual path delivers full-rank gradient signal to all encoder channels even though only the first $z = Wx$7-dimensional slice is supervised, thereby preventing collapse (see Figure 1 in (Jing et al., 2021)).

3. Network Architecture and Training Protocol

• Backbone: ResNet50 encoder producing a 2048-dimensional representation.
• Projector: None; the first $z = Wx$8 channels are extracted as features.
• Loss: InfoNCE is computed on the $z = Wx$9-normalized $$\dot W = -W X \qquad X = \hat\Sigma_0 - \hat\Sigma_1,$$0-dimensional slice.
• Augmentations: Random crop and resize to 224×224, color jitter, grayscale, Gaussian blur, solarization, horizontal flip (matching SimCLR).
• Optimizer: LARS, base learning rate $$\dot W = -W X \qquad X = \hat\Sigma_0 - \hat\Sigma_1,$$1 (scaled by batch), 10-epoch warmup, cosine decay schedule over 100 epochs.
• Batch size: 4096 samples, distributed over 32 GPUs.
• Key hyperparameter: $$\dot W = -W X \qquad X = \hat\Sigma_0 - \hat\Sigma_1,$$2; experimentally tuned within $$\dot W = -W X \qquad X = \hat\Sigma_0 - \hat\Sigma_1,$$3, with optimal performance typically near $$\dot W = -W X \qquad X = \hat\Sigma_0 - \hat\Sigma_1,$$4.

Schematic pseudo-code: $\hat\Sigma_0$1

4. Empirical Evaluation and Comparative Analysis

The principal empirical benchmark is linear-probe Top-1 accuracy on ImageNet after 100 epochs. The following table summarizes the main results:

Model	Top-1 Acc (%)
SimCLR 2-layer nonlinear projector	66.5
SimCLR 1-layer linear projector	61.1
SimCLR no projector	51.5
DirectCLR (no proj; $$\dot W = -W X \qquad X = \hat\Sigma_0 - \hat\Sigma_1,$$5)	62.7

DirectCLR yields +1.6 percentage points over SimCLR with a single-layer linear projector and recovers most of the gap to the full 2-layer MLP.

Ablation analysis demonstrates (Table 2) that:

Projector	Diagonal	Low-rank	Top-1 (%)
none	—	—	51.5
orthogonal (fixed 1’s)	—	—	52.2
trainable (full linear)	—	—	61.1
trainable diagonal	✓	—	60.2
fixed low-rank (rand. orthogonal)	—	✓	62.3
fixed low-rank diagonal	✓	✓	62.7

Thus, for projector effectiveness, only the singular-value spectrum (diagonal structure) and low-rank-ness are necessary; DirectCLR’s fixed sliced subvector provides both properties by construction.

Analysis of singular value spectra (Figure 2) reveals that DirectCLR’s representations are nearly as full-rank as SimCLR with a projector, and far less collapsed than SimCLR without one.

5. Practical Implementation Guidance

• Employ the same data-augmentation and optimizer schedules as SimCLR, including LARS, large batch size, and cosine LR decay.
• Tuning $$\dot W = -W X \qquad X = \hat\Sigma_0 - \hat\Sigma_1,$$6 is fundamental: if $$\dot W = -W X \qquad X = \hat\Sigma_0 - \hat\Sigma_1,$$7 is too small, insufficient gradient is propagated; if too large, dimensional collapse recurs as in SimCLR without a projector. $$\dot W = -W X \qquad X = \hat\Sigma_0 - \hat\Sigma_1,$$8 values between 256 and 512 perform well, with peak validation accuracy around $$\dot W = -W X \qquad X = \hat\Sigma_0 - \hat\Sigma_1,$$9 (see Figure 3).
• Use a fixed slice: always select the first $\hat\Sigma_0$0 channels of the encoder; stochastic slicing severely degrades performance (43% Top-1).
• Training regime: 100 epochs on ImageNet, as in SimCLR, for direct comparability.

DirectCLR represents a minimal-parameter, theoretically motivated alternative to MLP projectors in contrastive SSL, directly leveraging the analysis of dimensional collapse by restricting and supervising a fixed, low-rank sub-block of the encoder output. This approach preserves most of the downstream performance of multi-layer projectors and sharply outperforms projectorless SimCLR, validating the theoretical claims with experimental results (Jing et al., 2021).