SAG-VICReg: Stable & Global SSL

• SAG-VICReg is a self-supervised representation learning framework that extends VICReg by incorporating semantically meaningful cross-image pairs using a random-walk mechanism.
• It replaces the standard instance-level invariance with a weighted invariance term that promotes global semantic structure in the embedding space.
• Empirical evaluations on datasets like ImageNet and CIFAR-100 show improved hierarchical and global structure metrics compared to traditional VICReg.

SAG-VICReg (Stable and Generalizable VICReg) is a framework for self-supervised representation learning that modifies the Variance-Invariance-Covariance Regularization (VICReg) objective by injecting semantically meaningful cross-image positive pairs using a random-walk pairing mechanism. This approach targets both improved generalization to unseen data and enhanced capture of global semantic structures in the learned embeddings, addressing structural limitations inherent in VICReg's original formulation (Simai et al., 22 Jun 2025).

1. Background: VICReg and Spectral Embedding View

VICReg is a self-supervised learning (SSL) algorithm that learns image representations by optimizing three terms:

• Invariance:

$$s(Y,Y') = \frac{1}{n} \sum_{i=1}^n \|y_i - y_i'\|_2^2$$

This term enforces that two random augmentations of the same image produce similar embeddings.

• Variance:

$$v(Y) = \frac{1}{k} \sum_{j=1}^k \max\left(0, \gamma - S(y^j, \epsilon)\right), \ \ S(y^j,\epsilon)=\sqrt{\mathrm{Var}(y^j) + \epsilon}$$

It ensures a minimal spread in each embedding dimension, preventing collapse.

• Covariance:

$c(Y) = \frac{1}{k} \sum_{p \neq q} [C(Y)]_{pq}^2$

This encourages decorrelation across embedding dimensions.

The combined loss function is: $$L_{\rm VICReg}(Y,Y') = \lambda s(Y,Y') + \mu\left[v(Y) + v(Y')\right] + \nu\left[c(Y) + c(Y')\right]$$

VICReg can be interpreted as spectral embedding on a block-diagonal graph where edges connect only augmentations of the same image. Consequently, it does not capture global relationships between images, leading to sub-optimal representation of unseen data clusters and potentially distorted embeddings for out-of-distribution samples. This view is analogous to SpectralNet objectives but with the restriction that the graph only encodes instance-level locality.

2. Random-Walk Pairing: Injecting Global Structure

To overcome the absence of cross-image relationships in VICReg, SAG-VICReg extends the local, block-diagonal neighborhood by constructing semantically-informed, stochastic cross-image pairings using random walks on a learned affinity graph. The procedure is as follows:

1. Generate two batches of embeddings:

$Z = \{z_i\}_{i=1}^n,\quad Z' = \{z_i'\}_{i=1}^n$

Each derived from random augmentations.

2. Build an affinity matrix $W \in \mathbb{R}^{n \times n}$ based on pairwise cosine distances between $z_i$ and $z_j'$:

$d_{ij} = 1 - \cos(z_i, z_j')$

For each $i$, retain $$v(Y) = \frac{1}{k} \sum_{j=1}^k \max\left(0, \gamma - S(y^j, \epsilon)\right), \ \ S(y^j,\epsilon)=\sqrt{\mathrm{Var}(y^j) + \epsilon}$$0 nearest neighbors (typically $$v(Y) = \frac{1}{k} \sum_{j=1}^k \max\left(0, \gamma - S(y^j, \epsilon)\right), \ \ S(y^j,\epsilon)=\sqrt{\mathrm{Var}(y^j) + \epsilon}$$1), and set a local scale $$v(Y) = \frac{1}{k} \sum_{j=1}^k \max\left(0, \gamma - S(y^j, \epsilon)\right), \ \ S(y^j,\epsilon)=\sqrt{\mathrm{Var}(y^j) + \epsilon}$$2 as the 20th percentile of $$v(Y) = \frac{1}{k} \sum_{j=1}^k \max\left(0, \gamma - S(y^j, \epsilon)\right), \ \ S(y^j,\epsilon)=\sqrt{\mathrm{Var}(y^j) + \epsilon}$$3:

$$v(Y) = \frac{1}{k} \sum_{j=1}^k \max\left(0, \gamma - S(y^j, \epsilon)\right), \ \ S(y^j,\epsilon)=\sqrt{\mathrm{Var}(y^j) + \epsilon}$$4

3. Define the normalized transition matrix for a one-step random walk:

$$v(Y) = \frac{1}{k} \sum_{j=1}^k \max\left(0, \gamma - S(y^j, \epsilon)\right), \ \ S(y^j,\epsilon)=\sqrt{\mathrm{Var}(y^j) + \epsilon}$$5

$$v(Y) = \frac{1}{k} \sum_{j=1}^k \max\left(0, \gamma - S(y^j, \epsilon)\right), \ \ S(y^j,\epsilon)=\sqrt{\mathrm{Var}(y^j) + \epsilon}$$6 gives the probability of transitioning from $$v(Y) = \frac{1}{k} \sum_{j=1}^k \max\left(0, \gamma - S(y^j, \epsilon)\right), \ \ S(y^j,\epsilon)=\sqrt{\mathrm{Var}(y^j) + \epsilon}$$7 to $$v(Y) = \frac{1}{k} \sum_{j=1}^k \max\left(0, \gamma - S(y^j, \epsilon)\right), \ \ S(y^j,\epsilon)=\sqrt{\mathrm{Var}(y^j) + \epsilon}$$8.

1. Sample for each $$v(Y) = \frac{1}{k} \sum_{j=1}^k \max\left(0, \gamma - S(y^j, \epsilon)\right), \ \ S(y^j,\epsilon)=\sqrt{\mathrm{Var}(y^j) + \epsilon}$$9 an index $c(Y) = \frac{1}{k} \sum_{p \neq q} [C(Y)]_{pq}^2$0 and set $c(Y) = \frac{1}{k} \sum_{p \neq q} [C(Y)]_{pq}^2$1. The new batch $c(Y) = \frac{1}{k} \sum_{p \neq q} [C(Y)]_{pq}^2$2 comprises these cross-image positive pairs.
2. Define a weighted invariance term:

$c(Y) = \frac{1}{k} \sum_{p \neq q} [C(Y)]_{pq}^2$3

(In implementation, one $c(Y) = \frac{1}{k} \sum_{p \neq q} [C(Y)]_{pq}^2$4 is sampled per $c(Y) = \frac{1}{k} \sum_{p \neq q} [C(Y)]_{pq}^2$5; the expectation matches the above expression.)

This process integrates semantically related cross-image pairs into the invariance objective, enriching the geometry underlying the embedding space.

3. The SAG-VICReg Objective

The loss function for SAG-VICReg retains the original variance and covariance regularization but replaces instance-only invariance with random-walk weighted invariance. The objective is:

$c(Y) = \frac{1}{k} \sum_{p \neq q} [C(Y)]_{pq}^2$6

or, grouping terms,

$c(Y) = \frac{1}{k} \sum_{p \neq q} [C(Y)]_{pq}^2$7

with the canonical hyperparameters $c(Y) = \frac{1}{k} \sum_{p \neq q} [C(Y)]_{pq}^2$8, $c(Y) = \frac{1}{k} \sum_{p \neq q} [C(Y)]_{pq}^2$9, $$L_{\rm VICReg}(Y,Y') = \lambda s(Y,Y') + \mu\left[v(Y) + v(Y')\right] + \nu\left[c(Y) + c(Y')\right]$$0.

The inclusion of $$L_{\rm VICReg}(Y,Y') = \lambda s(Y,Y') + \mu\left[v(Y) + v(Y')\right] + \nu\left[c(Y) + c(Y')\right]$$1 in place of instance-only $$L_{\rm VICReg}(Y,Y') = \lambda s(Y,Y') + \mu\left[v(Y) + v(Y')\right] + \nu\left[c(Y) + c(Y')\right]$$2 promotes invariance not just among multiple views of the same image, but also across semantically proximate images, directly enhancing global semantic structure in the learned representation.

4. Standalone Evaluation Metric for Global Structure

To assess embedding quality on global semantic structure without requiring labels, SAG-VICReg introduces a dendrogram-based similarity metric:

• Use agglomerative clustering (Ward linkage, cosine distance) to construct dendrograms $$L_{\rm VICReg}(Y,Y') = \lambda s(Y,Y') + \mu\left[v(Y) + v(Y')\right] + \nu\left[c(Y) + c(Y')\right]$$3 and $$L_{\rm VICReg}(Y,Y') = \lambda s(Y,Y') + \mu\left[v(Y) + v(Y')\right] + \nu\left[c(Y) + c(Y')\right]$$4 for two embedding sets $$L_{\rm VICReg}(Y,Y') = \lambda s(Y,Y') + \mu\left[v(Y) + v(Y')\right] + \nu\left[c(Y) + c(Y')\right]$$5 and $$L_{\rm VICReg}(Y,Y') = \lambda s(Y,Y') + \mu\left[v(Y) + v(Y')\right] + \nu\left[c(Y) + c(Y')\right]$$6 (e.g., train and test splits).
• Compute Lowest Common Ancestor (LCA) distances for all pairs $$L_{\rm VICReg}(Y,Y') = \lambda s(Y,Y') + \mu\left[v(Y) + v(Y')\right] + \nu\left[c(Y) + c(Y')\right]$$7: the dendrogram height where pair $$L_{\rm VICReg}(Y,Y') = \lambda s(Y,Y') + \mu\left[v(Y) + v(Y')\right] + \nu\left[c(Y) + c(Y')\right]$$8 merges in $$L_{\rm VICReg}(Y,Y') = \lambda s(Y,Y') + \mu\left[v(Y) + v(Y')\right] + \nu\left[c(Y) + c(Y')\right]$$9 yields $Z = \{z_i\}_{i=1}^n,\quad Z' = \{z_i'\}_{i=1}^n$0 and analogously for $Z = \{z_i\}_{i=1}^n,\quad Z' = \{z_i'\}_{i=1}^n$1.
• Calculate Pearson, Spearman, and Kendall correlations between $Z = \{z_i\}_{i=1}^n,\quad Z' = \{z_i'\}_{i=1}^n$2 and $Z = \{z_i\}_{i=1}^n,\quad Z' = \{z_i'\}_{i=1}^n$3; high rank correlation indicates similar hierarchical structure.
• Assess cophenetic correlation via:

$Z = \{z_i\}_{i=1}^n,\quad Z' = \{z_i'\}_{i=1}^n$4

where $Z = \{z_i\}_{i=1}^n,\quad Z' = \{z_i'\}_{i=1}^n$5 are the original distances, $Z = \{z_i\}_{i=1}^n,\quad Z' = \{z_i'\}_{i=1}^n$6 are dendrogram distances.

This metric directly quantifies the preservation of global structure across domains or data splits in the absence of labels.

5. Algorithm and Hyperparameters

The SAG-VICReg procedure can be summarized as follows:

1. Apply two random augmentations to each batch sample, yielding $Z = \{z_i\}_{i=1}^n,\quad Z' = \{z_i'\}_{i=1}^n$7.
2. Compute representations $Z = \{z_i\}_{i=1}^n,\quad Z' = \{z_i'\}_{i=1}^n$8 and embeddings $Z = \{z_i\}_{i=1}^n,\quad Z' = \{z_i'\}_{i=1}^n$9.
3. Form batches $W \in \mathbb{R}^{n \times n}$0.
4. Compute pairwise cosine distances $W \in \mathbb{R}^{n \times n}$1.
5. For each $W \in \mathbb{R}^{n \times n}$2, construct local scale $W \in \mathbb{R}^{n \times n}$3 from the 20th percentile of $W \in \mathbb{R}^{n \times n}$4 and compute affinity $W \in \mathbb{R}^{n \times n}$5 over $W \in \mathbb{R}^{n \times n}$6 nearest neighbors.
6. Normalize $W \in \mathbb{R}^{n \times n}$7 to transition matrix $W \in \mathbb{R}^{n \times n}$8.
7. For each $W \in \mathbb{R}^{n \times n}$9, sample $z_i$0, set $z_i$1, yielding $z_i$2.
8. Calculate weighted invariance $z_i$3, variances, and covariances on $z_i$4 and $z_i$5.
9. Compute total loss: $z_i$6. 10. Backpropagate gradients and update parameters.

Hyperparameters: $z_i$7, $z_i$8, $z_i$9, batch size $z_j'$0, random-walk neighbors $z_j'$1, local scale via 20th percentile.

6. Empirical Results and Benchmarks

Evaluations on ImageNet-1k, CIFAR-100, and Caltech-256 reveal that SAG-VICReg consistently surpasses or matches VICReg and state-of-the-art SSL baselines on both label-free and conventional structure-preserving metrics:

• LCA Similarity (Spearman): SAG-VICReg achieves 0.433 on ImageNet versus VICReg's 0.403 (+7%), and 0.296 on CIFAR-100 versus VICReg's 0.223 (+33%).
• Cophenetic similarity: On CIFAR-100 D1$z_j'$2P2, SAG-VICReg yields 0.405 compared to the next best 0.349 (+16%).
• Hierarchical Rand Index: Superior results at all hierarchy levels, with larger gains at intermediate granularity.
• Hierarchical $z_j'$3-NN and linear probes: Best or competitive performance on both coarse and fine levels, indicating no compromise in local discriminability.
• Masking-based comparisons on CIFAR-100: Outperforms DINO, MAE, and I-JEPA in rank correlations despite differing architectural design philosophies.

These results indicate that the random-walk pairing mechanism meaningfully enhances global semantic coherence and stability of the learned representations, an effect captured distinctly by the proposed global-structure metric.

7. Significance and Implications

By integrating a principled random-walk–based extension over the original block-diagonal affinity graph of VICReg, SAG-VICReg addresses limitations in generalization and semantic fidelity for self-supervised representations. The approach leverages spectral embedding theory to diagnose sub-optimalities and prescribes an efficient and differentiable modification applicable in current deep learning workflows. The dendrogram-based metric facilitates objective, label-free benchmarking for global embedding structure, of practical value where annotated data is unavailable. A plausible implication is that random-walk pairing or similar spectral enrichment mechanisms may serve as a general template for augmenting contrastive and non-contrastive SSL objectives to better capture data manifold geometry, irrespective of domain (Simai et al., 22 Jun 2025).