Self-Supervised Loss Formulation

• Self-supervised loss formulation is the design of objective functions that leverage intrinsic data signals and surrogate tasks to train models without direct labels.
• The approach integrates contrastive, non-contrastive, probabilistic, and geometric methodologies to optimize representations and maintain invariance to specific data transformations.
• Practical implementation requires careful tuning of data augmentations and hyperparameters, ensuring robustness across domains such as vision, speech, and sequential processing.

Self-supervised loss formulation refers to the mathematical and algorithmic design of objective functions used to train models on unlabeled data by leveraging invariance, reconstruction, deformation, or instance-discriminative signals derived from intrinsic data structure or transformations. These losses lie at the core of self-supervised learning (SSL) across domains such as vision, speech, and sequential processing, and enable the learning of general-purpose representations by exploiting surrogate tasks—without direct access to ground truth labels. Loss formulation defines both the signal to which the model becomes invariant or sensitive and the operational regime—contrastive, non-contrastive, redundancy-reducing, uncertainty-aware, geometric, or physical—within which representations are optimized.

1. Theoretical Foundations and Probabilistic Models

Modern SSL loss design is increasingly grounded in probabilistic and information-theoretic frameworks. A notable example is the latent-variable generative model for non-contrastive SSL by Fleissner et al. (Fleissner et al., 22 Jan 2025), where representations are formally tied to maximum-likelihood estimation of a hierarchy:

• Latent variable $z \sim \mathcal{N}(0, I_k)$; observed anchor $x \mid z \sim \mathcal{N}(Wz, A)$; and positive (augmented) view $x^+ \mid x \sim \mathcal{N}(x, B)$ with positive semi-definite covariances $A$, $B$.
• The negative log-likelihood

$$L(\theta) = \log\det \Sigma + \operatorname{Tr}(\Sigma^{-1} S)$$

with block covariance $\Sigma$ and empirical scatter $S$ directly relates the SSL loss to classical PCA when augmentations are uninformative ($B\propto I$), and to a pull-together squared-difference loss when augmentations preserve the signal subspace (orthogonal noise).

• This model elucidates that the informativeness of data augmentations—determined by the alignment of $B$ with the subspace $W$—governs the type of representation recovered, seamlessly interpolating between principal component analysis and non-contrastive alignment losses.
• The Bayesian extension equates to MAP estimation, permitting priors on $W$ for uncertainty-aware representation learning.

This rigorous analysis underlines that the desired invariances in SSL losses are fundamentally linked to the geometric properties imposed by augmentation covariances and that poor augmentation choices can cause SSL to collapse to trivial variance-maximization (Fleissner et al., 22 Jan 2025).

2. Loss Taxonomy: Contrastive, Non-Contrastive, Margin-based, and Geometric Formulations

2.1. Contrastive and Margin-based Objectives

Contrastive losses, typified by InfoNCE/NT-Xent, train encoders to maximize similarities within positive pairs and minimize them across negatives:

$$\mathcal{L}_{\mathrm{NT{-}Xent}} = -\frac{1}{N}\sum_{i=1}^N \log \frac{\exp(\cos(z_i, z_i')/\tau)} {\sum_{a=1}^N \exp(\cos(z_i, z_a')/\tau)}$$

where $\tau$ is the temperature (Lepage et al., 2023). Symmetric variants (SNT-Xent) double the supervisory signal by treating all views as anchors, enhancing gradient uniformity.

Additive Margin (AM-Softmax) and Additive Angular Margin (AAM-Softmax) inject explicit margins to separate positives from negatives:

• AM: $$\ell^+_{\mathrm{AM}}(u, v) = \exp\frac{\cos(u, v) - m}{\tau}$$
• AAM: $$\ell^+_{\mathrm{AAM}}(u, v) = \exp\frac{\cos(\arccos(u^\top v) + m)}{\tau}$$ These variants shrink intra-class variance and widen inter-class boundaries, and their integration into SSL loss design yields improvements in tasks such as self-supervised speaker verification (Lepage et al., 2023). Hyperparameters $m$ (margin) and $\tau$ are tuned jointly for optimal separation and stability.

Angular Contrastive Loss (ACL) adds a global angular margin constraint explicitly:

$$L_A(i, j) = \begin{cases} [\arccos\langle h_i, h_j\rangle]^2, & \text{if positive} \ \max(0, m_g - \arccos\langle h_i, h_j\rangle )^2, & \text{if negative} \end{cases}$$

with the total loss $ACL = \alpha L_C + (1-\alpha) L_A$ interpolating between classic NT-Xent and angular exclusion (Wang et al., 2022).

2.2. Non-Contrastive, Redundancy Reduction, and Pseudo-Whitening

Non-contrastive losses, such as those analyzed in (Fleissner et al., 22 Jan 2025), directly penalize the discrepancy between features of positive pairs, e.g., via squared Euclidean or MSE losses, and are closely related to whitening or redundancy reduction approaches (Barlow Twins, VICReg).

GUESS (Mohamadi et al., 2024) generalizes this by incorporating data-driven "uncertainty" from generative autoencoder branches into the whitening constraint:

$$\mathcal{L}_w = \sum_i (1 - C_{ii})^2 + \beta \sum_{i \neq j} (C_{ij} - C^{(E)}_{ij})^2$$

Here, $C$ is the embedding cross-correlation, $C^{(E)}$ that of the autoencoder's latent codes; this design allows the network to preserve or suppress off-diagonal correlations based on empirical data variability, preventing the hard invariance of classic whitening and supporting richer, uncertainty-aware representations.

3. Information-Theoretic and Probabilistic Extensions

Explicit mutual information maximization is the theoretically optimal SSL criterion but challenging to realize in high-dimensional settings. Recent advances provide tractable formulations:

• Under the assumption that embedding distributions are marginally homeomorphic to Gaussians, MI between paired representations is

$$I(Z; Z') = \log \frac{\det C_{[Z, Z']}}{\det C_{ZZ} \det C_{Z'Z'}}$$

where $C_{[Z, Z']}$ is the joint covariance. The corresponding loss

$$\mathcal{L}_{\mathrm{MI}} = \log\det(C_{ZZ} - C_{ZZ'}) - \log\det C_{ZZ} - \log\det C_{Z'Z'}$$

can be effectively estimated with batchwise Gram matrices and stabilized using truncated Taylor expansions and dynamic rescaling (Chang et al., 2024). This loss avoids negative sampling, inherently decorrelates features, and is an unbiased estimator under homeomorphic distributions.

• Probabilistic contrastive loss extends NT-Xent by embedding views into r-radius von Mises–Fisher distributions with per-sample confidence $\kappa$,

$$s(x_i, x_j) = \log C_d(\kappa_i) + \log C_d(\kappa_j) - \log C_d(\tilde{\kappa}) - d \log r$$

where $$\tilde{\kappa} = \|\kappa_i \mu_i + \kappa_j \mu_j\|$$ is a combined concentration, and $C_d$ is the vMF normalizer. This enables soft attention on uncertain pairs, and the induced loss contracts or dilutes the alignment based on confidence (Li et al., 2021).

4. Task-Structured, Geometric, and Physics-Informed Losses

Beyond generic representation learning, SSL losses can encode domain knowledge by integrating geometric or physical constraints.

• In visual localization and depth estimation, joint losses couple photometric consistency, geometric (depth/epipolar) consistency, and sequence-wise constraints:

$$\mathcal{L} = \frac{1}{|\mathcal{V}|} \sum_{p \in \mathcal{V}} [ \min_{s \neq t}\mathcal{C}_s(p) + \lambda_{\mathrm{smooth}} \mathcal{L}_{\mathrm{smooth}}(p) ]$$

where $\mathcal{C}_s(p)$ combines robust SSIM+L1 photometric terms and normalized depth-consistency across frames, and minimum-cost selection across sliding windows forces the model to resolve scene geometry from long-range multiview context (Xu et al., 23 Jan 2026). Isometric self-sample based learning synthesizes new views via known static 3D transforms, enforcing scale-aligned depth consistency without dynamic region masking (Cha et al., 2022).

• In pose prediction from RGB+polarimetric data (Ruhkamp et al., 2023), self-supervised loss consists of a pseudo-label loss (rendered mask and normal losses coupled to the teacher pose) and a physical inconsistency term measuring degree-of-polarization error under forward-inverted physical models. The invertible constraint ensures predictions generate surface normals compatible with raw polarimetric observations, forgoing ground-truth pose entirely.

5. Domain-Specific and Auxiliary Self-Supervised Losses

Self-supervised objectives are often tailored to, or combined with, domain-structured pretext tasks:

• In speech enhancement, losses are defined by the MSE between enhanced output and clean utterance in the feature space of a frozen self-supervised model (HuBERT, XLSR)—either at the convolutional encoder or transformer output stage. Early-layer features, rather than deeper representations, correlate best with perceptual metrics (PESQ, STOI, and MOS) (Close et al., 2023, Close et al., 2023).
• For transfer learning and continual learning, SSL losses are adapted into knowledge distillation pipelines, where a small predictor learns to regress current representations to a frozen past model state, applying any SSL objective originally designed for static training (Fini et al., 2021). In few-shot regimes, auxiliary pretext tasks such as rotation prediction and jigsaw puzzle classification regularize encoded features and improve generalization (Su et al., 2019).

6. Connections to Supervised Objectives and Balance of Attraction–Repulsion

A growing theoretical literature analyzes SSL losses as asymptotic or instance-level proxies for supervised embeddings:

• Supervised contrastive losses naturally separate “attraction” to class prototypes from “repulsion” from others. Balanced contrastive losses generalize InfoNCE by introducing softmax-weighted repelling terms and a tunable balance parameter $\lambda$:

$$\ell_{\mathrm{BCL}}(z) = -s(z, z^+) + \lambda \left[ \frac{1}{\alpha}\log \sum_{z^-} e^{\alpha s(z, z^-)} \right ]$$

Adjusting $\lambda$ controls the spectrum from strong uniformity to strong clustering, while the sharpness parameter $\alpha$ modulates focus on hardest negatives (Lee, 12 Oct 2025). Decoupling the positive pair in denominator aligns with recent findings in decoupled contrastive learning.

• In supervised settings, SINCERE loss (Feeney et al., 2023) exposes that including same-class positives as negatives induces intra-class repulsion, which is eliminated by correct form; the resulting loss is tightly lower-bounded by the symmetrized KL divergence between true and false class distributions, yielding embedding spaces with improved separation and transfer qualities.

7. Empirical Design, Implementation, and Best Practices

Critical operational choices include:

• Selection and informativeness of data augmentations, which must be tuned to preserve semantic signals rather than inducing collapses to variance-dominated or noise-dominated regimes (Fleissner et al., 22 Jan 2025, Lee, 12 Oct 2025).
• Hyperparameter sensitivity: margin magnitude $m$ (AM/AAM), angular thresholds, attractiveness–repulsiveness balance $\lambda$ and sharpness $\alpha$, and temperature $\tau$ necessitate empirical optimization specific to task and modality (Lepage et al., 2023, Wang et al., 2022).
• Modalities such as speech benefit from matching the pretraining domain of the self-supervised representation extractor to the enhancement or recognition domain whenever possible (Close et al., 2023).
• In redundancy-reduction or whitening approaches, tuning the relaxation-to-target covariances and leveraging uncertainty-induced targets (e.g., GUESS) significantly enhances robustness to augmentation variability (Mohamadi et al., 2024).

Widely adopted implementation patterns include: batch-based Gram or covariance estimation with rescaling (for moment-based losses), curriculum-scheduled margins, symmetric view treatment, auxiliary task heads (rotation, jigsaw), continual distillation via frozen model replay, and per-layer or per-channel normalization.

• (Fleissner et al., 22 Jan 2025) A Probabilistic Model for Non-Contrastive Learning
• (Lepage et al., 2023) Experimenting with Additive Margins for Contrastive Self-Supervised Speaker Verification
• (Wang et al., 2022) Self-supervised learning of audio representations using angular contrastive loss
• (Mohamadi et al., 2024) GUESS: Generative Uncertainty Ensemble for Self Supervision
• (Chang et al., 2024) Explicit Mutual Information Maximization for Self-Supervised Learning
• (Li et al., 2021) Probabilistic Contrastive Loss for Self-Supervised Learning
• (Cha et al., 2022) Self-Supervised Depth Estimation with Isometric-Self-Sample-Based Learning
• (Xu et al., 23 Jan 2026) GPA-VGGT: Adapting VGGT to Large scale Localization by self-Supervised learning with Geometry and Physics Aware loss
• (Ruhkamp et al., 2023) S2P3: Self-Supervised Polarimetric Pose Prediction
• (Feeney et al., 2023) SINCERE: Supervised Information Noise-Contrastive Estimation REvisited
• (Lee, 12 Oct 2025) Understanding Self-supervised Contrastive Learning through Supervised Objectives
• (Fini et al., 2021) Self-Supervised Models are Continual Learners
• (Close et al., 2023) Perceive and predict: self-supervised speech representation based loss functions for speech enhancement
• (Close et al., 2023) The Effect of Spoken Language on Speech Enhancement using Self-Supervised Speech Representation Loss Functions
• (Su et al., 2019) Boosting Supervision with Self-Supervision for Few-shot Learning

These references establish the breadth of contemporary research in self-supervised loss formulation, ranging from foundational probabilistic and geometric analysis to domain-specific integration and practical architectural considerations.