Gram Anchoring Regularization

• Gram anchoring regularization is a technique that preserves second-order feature statistics by aligning Gram matrices across domains or training snapshots.
• It employs distinct angular and scale loss components to mitigate feature drift, ensuring robust generalization in unsupervised domain adaptation and self-supervised vision models.
• Empirical results indicate significant improvements, including reduced MAE in regression tasks and recovered mIoU in vision transformers, highlighting its practical benefit.

Gram anchoring regularization refers to a family of explicit regularizers that align or preserve Gram matrices—typically, sample-wise or patch-wise similarity matrices derived from network feature representations—during training. By penalizing discrepancies between selected Gram matrices, these regularizers constrain important geometric or statistical properties of the feature space. Recent work demonstrates the efficacy of Gram anchoring in both unsupervised domain adaptation for regression and long-run self-supervised vision training. This technique systematically addresses feature drift and subspace discrepancies, leading to more robust generalization and dense prediction performance (Nejjar et al., 2023, Siméoni et al., 13 Aug 2025).

1. Conceptual Foundations and Motivation

Gram anchoring regularization enforces consistency between the second-order statistics (i.e., Gram matrices) of feature embeddings—either across domains, between snapshots, or between teacher and student networks. The rationale for this approach arises in two distinct but related problem settings:

• Unsupervised Domain Adaptation Regression (UDA-Reg): In DARE-GRAM, standard feature-space alignment does not guarantee that the optimal Ordinary Least Squares (OLS) solution for a linear regressor will transfer across domains, because the OLS regressor explicitly depends on the inverse Gram matrix of features. Aligning feature means or first-order statistics alone leaves the possibility of significant domain shift in the Gram (second-order) structure, resulting in suboptimal transfer (Nejjar et al., 2023).
• Self-Supervised Learning for ViTs: In DINOv3, long-duration self-supervised training of transformer backbones leads to progressive collapse or drift in local patchwise feature similarities, degrading dense downstream performance. Anchoring the student's Gram matrix of patch embeddings to that of an earlier “Gram teacher” network snapshot preserves the fine-grained geometric structure of the latent space, preventing the loss of local information (Siméoni et al., 13 Aug 2025).

2. Mathematical Formulations

While differing in purpose and technical detail, both DARE-GRAM and DINOv3 define Gram anchoring loss terms as penalized distances between Gram matrices:

DARE-GRAM (Domain Adaptation)

Let $X_s, X_t \in \mathbb{R}^{b \times p}$ be mini-batch feature matrices for source and target, with Gram matrices $G_s = X_s^\top X_s$, $G_t = X_t^\top X_t$. Truncated pseudo-inverses $G_s^+, G_t^+$ are computed in a $k$-dimensional principal subspace. Two distinct loss terms are defined:

• Angular Alignment: For each column $i$, measure the cosine similarity between $g_{s,i}^+$ and $g_{t,i}^+$ (columns of $G_s^+, G_t^+$), defining

$L_{\text{angle}} = \sum_{i=1}^{p} |1 - m_i|$

with $$m_i = \frac{g_{s,i}^+ \cdot g_{t,i}^+}{\|g_{s,i}^+\|_2 \|g_{t,i}^+\|_2}$$.

• Scale Alignment: Denote top-$k$ eigenvalues $\lambda_{s,1},\dots,\lambda_{s,k}$ (source) and $\lambda_{t,1},\dots,\lambda_{t,k}$ (target):

$$L_{\text{scale}} = \left\| [\lambda_{s,1\ldots k}] - [\lambda_{t,1\ldots k}] \right\|_2^2$$

The total Gram-anchoring loss is $$L_{\text{gram}} = \alpha L_{\text{angle}} + \beta L_{\text{scale}}$$ (Nejjar et al., 2023).

DINOv3 (Self-Supervised Learning)

Given normalized patch embedding matrices $X_S \in \mathbb{R}^{P \times d}$ from the student and $X_G \in \mathbb{R}^{P \times d}$ from the Gram teacher (previous-EMA snapshot), Gram matrices are $G_S = X_S X_S^\top$ and $G_G = X_G X_G^\top$. The Gram anchoring loss is:

$$L_{\text{Gram}} = \| G_S - G_G \|_F^2 = \| X_S X_S^\top - X_G X_G^\top \|_F^2$$

This loss is applied to global crops only, with the Gram teacher updated every $\Delta T$ steps (Siméoni et al., 13 Aug 2025).

3. Algorithmic Integration and Optimization

DARE-GRAM Training Workflow

1. Sample batches from source (labeled) and target (unlabeled) datasets.
2. Compute feature embeddings $Z_s = h_\theta(x_s)$, $Z_t = h_\theta(x_t)$.
3. Compute source regression loss $$L_{\text{src}} = \frac{1}{b} \sum_{i=1}^{b} \| \hat{y}_s^i - y_s^i \|_2^2$$.
4. Calculate Gram matrices $G_s, G_t$, their SVDs, truncate to a principal subspace, and compute pseudo-inverses $G_s^+, G_t^+$.
5. Compute $L_{\text{angle}}$ and $L_{\text{scale}}$ as above.
6. Total loss: $$L_{\text{total}} = L_{\text{src}} + \alpha L_{\text{angle}} + \beta L_{\text{scale}}$$.
7. Update encoder and regressor via backpropagation (Nejjar et al., 2023).

Subspace Selection:

The principal subspace is chosen such that cumulative explained variance $T$ is reached (e.g., $T \approx 0.95$). The pseudo-inverse is formed only over the retained components, mitigating numerical instability (Nejjar et al., 2023).

DINOv3 Gram Anchoring Workflow

1. Pretrain in standard SSL regime (DINO, iBOT, Koleo losses).
2. At iteration $T_0$, begin refinement: introduce $L_{\text{Gram}}$ anchored to a Gram teacher (EMA snapshot every $\Delta T = 10\text{k}$ steps).
3. For each global image crop:
   • Extract student and teacher patch embeddings.
   • Compute Gram matrices and $L_{\text{Gram}}$.
   • Total loss: $$w_D L_{\text{DINO}} + L_{\text{iBOT}} + 0.1 L_{\text{Koleo}} + w_G L_{\text{Gram}}$$.
4. Backpropagate, update student and EMA teacher (Siméoni et al., 13 Aug 2025).
5. The teacher is optionally computed at higher resolution for smoother Gram matrices.

4. Hyperparameters, Ablations, and Practical Recommendations

DARE-GRAM

• Alignment weights $(\alpha, \beta)$ are robust in $[10^{-2}, 10^{2}]$; common setting is $1.0$ each.
• Variance threshold for truncated subspace $T \in [0.8, 0.99]$, with $T \approx 0.95$ effective.
• Results are stable with respect to batch size and other training details (Nejjar et al., 2023).

DINOv3

• Gram-anchoring weight $w_G = 2$, DINO global loss weight $w_D = 1$.
• Teacher is updated every $10\text{k}$ steps.
• Dense refinement starts after $1$M iterations, following the peak of dense task metrics.
• High-resolution downsampled teachers and mid-training snapshots yield best results (Siméoni et al., 13 Aug 2025).
• Only global crops are regularized; overhead is modest, requiring extra memory for Gram teacher patch tensors and some $O(P^2 d)$ computations per crop.

5. Empirical Effects and Benchmark Results

DARE-GRAM

Experiments with dSprites, MPI3D, and Biwi Kinect (head-pose estimation) demonstrate consistent improvements in target domain mean absolute error (MAE):

Dataset	Baseline (ResNet-18)	RSD (subspace align.)	DARE-GRAM	Relative MAE Reduction
dSprites	0.498	0.237	0.164	30.8%
MPI3D	0.377	0.205	0.160	21.9%
Biwi Kinect	0.335	0.280	0.260	7.1%

Ablation studies confirm that aligning inverse Gram matrices is significantly more effective than aligning first-order features ($Z$), and that both scale and angular terms matter (Nejjar et al., 2023).

DINOv3

For dense vision benchmarks (e.g., Pascal-VOC):

• Without Gram anchoring, mIoU rises (to $\sim$53 at 200k iters), then decays (to $\sim$48 at 1M iters).
• After introducing Gram anchoring at 1M iters, mIoU recovers sharply (to $\sim$55 within 10k steps).
• High-res downsampled teachers from mid-training yield best results (e.g., mIoU $=55.7$) (Siméoni et al., 13 Aug 2025).

6. Theoretical Significance and Extensions

Gram anchoring regularization shifts focus from aligning raw features to constraining geometric and statistical invariants (Gram matrices) more directly linked to task-optimal solutions or representational integrity. In UDA-regression, this directly targets the OLS solution's dependency on the inverse Gram. In SSL for vision transformers, it stabilizes dense representations against drift observed in long training runs.

Both approaches are non-intrusive, computationally moderate, and compatible with existing architectures and objectives. By operating at the level of similarity statistics, Gram anchoring offers a general strategy to encode prior structural knowledge or prevent undesired feature collapse in deep learning pipelines.

7. Comparison of Gram Anchoring Usages

Aspect	DARE-GRAM (Nejjar et al., 2023)	DINOv3 (Siméoni et al., 13 Aug 2025)
Domain	UDA regression	Self-supervised vision (ViTs)
Alignment Target	Inverse Gram (pseudo-inverse, subspace)	Direct patch Gram matrices
Core Losses	Scale ($L_{\text{scale}}$) and angle ($L_{\text{angle}}$)	Frobenius distance to teacher
Regularization Goal	Robust OLS regressor transfer	Prevent dense feature drift
Benchmark Improvements	MAE reductions up to 30.8%	mIoU recovery $\sim$7 points

Gram anchoring regularization thus represents a principled and practically validated approach for enforcing consistency and invariance in both supervised adaptation and self-supervised learning scenarios.