Cause of dimensional collapse in contrastive self-supervised learning

Determine the cause of dimensional collapse in InfoNCE-based contrastive self-supervised learning, including the mechanisms responsible for collapse beyond the last-layer weight-norm effects analyzed in the paper.

Background

The paper analyzes dimensional collapse in contrastive self-supervised learning through the rank dynamics of encoder and projector representations. Its theoretical framework identifies several possible mechanisms, including weight-norm shrinkage, inter-layer spectral misalignment, and feature-covariance misalignment, while the proposed regularization primarily addresses weight-induced collapse in the final layer.

Despite these analyses, the conclusion explicitly states that the study of the cause of dimensional collapse remains for future work. The unresolved problem is therefore to establish a comprehensive causal account of dimensional collapse in the studied contrastive-learning architectures, rather than only characterizing or mitigating its observed effects.