Gram Anchoring Loss in Vision Models

• Gram Anchoring Loss is a class of loss functions that utilize Gram matrices to capture second-order statistics, enforcing structural consistency in neural representations.
• It is applied in both vision-language model prompt tuning and self-supervised vision transformer training to improve domain robustness and preserve patch-level similarity.
• Empirical evidence demonstrates notable gains in out-of-distribution accuracy, restored dense feature quality, and enhanced prediction metrics with minimal computational overhead.

Gram Anchoring Loss denotes a class of loss functions that exploit second-order statistical structure—specifically Gram matrices—to regularize feature representations and enhance the robustness or spatial consistency of neural models. Two distinct strands of Gram Anchoring Loss have emerged in recent literature: a cross-entropy-based second-order stream in Vision-LLM (VLM) prompt tuning (Chen et al., 5 Apr 2026) and a Gram-structure-preserving regularizer for self-supervised vision transformers in dense feature learning (Siméoni et al., 13 Aug 2025). Despite differing in methodological detail and role, both approaches share the central objective of leveraging Gram matrix-derived statistics to provide structural priors or consistency constraints.

1. Second-Order Priors and Motivation

First-order token features in vision backbones (e.g., ViTs) furnish spatially grounded, semantic representations but are highly sensitive to domain shifts, local noise, or extended self-supervised training. Second-order statistics—specifically Gram matrices formed from patch features—encode class-level texture, style, or inter-token relational structure that is stable across domains and more resilient to spurious disruptions.

In VLM adaptation, Gram Anchoring Loss augments classical prompt alignment by anchoring prompts to these second-order descriptors, enabling the learning of domain-invariant or style-robust representations (Chen et al., 5 Apr 2026). For long-horizon self-supervised learning (SSL), Gram Anchoring counteracts the collapse of dense patch similarity structure by explicitly aligning student features’ Gram matrices to those of a “teacher” reference (Siméoni et al., 13 Aug 2025). In both cases, the loss targets global or relational stability unattainable by first-order objectives alone.

2. Methodologies and Mathematical Formulation

2.1 Gram Anchoring in Prompt Learning (GAPL)

The GAPL framework introduces a Gram-Anchored Stream on top of the CLIP backbone as follows:

• Extract image patch tokens $F \in\mathbb{R}^{N\times d}$ (excluding CLS) from the visual encoder.
• Compute channel-wise Gram matrix: $$G = \frac{1}{N} F^\top F \in \mathbb{R}^{d\times d}$$.
• Extract the diagonal: $g_{\text{diag}} = \text{diag}(G)\in \mathbb{R}^d$.
• Stabilize range: $u = \log(g_{\text{diag}}+\epsilon)$, with $\epsilon$ small (e.g., $1\mathrm{e}{-6}$).
• Pass $u$ through an MLP $\Phi$ and sigmoid to get a gating vector $\Gamma = \sigma(\Phi(u))$.
• Modulate class prompt $w_c\in\mathbb{R}^d$: $$G = \frac{1}{N} F^\top F \in \mathbb{R}^{d\times d}$$0.
• Compute cosine logits against global feature $$G = \frac{1}{N} F^\top F \in \mathbb{R}^{d\times d}$$1, scaled by CLIP temperature $$G = \frac{1}{N} F^\top F \in \mathbb{R}^{d\times d}$$2: $$G = \frac{1}{N} F^\top F \in \mathbb{R}^{d\times d}$$3.
• Softmax produces $$G = \frac{1}{N} F^\top F \in \mathbb{R}^{d\times d}$$4.
• The Gram Anchoring Loss is simply the cross-entropy applied along this stream:

$$G = \frac{1}{N} F^\top F \in \mathbb{R}^{d\times d}$$5

where $$G = \frac{1}{N} F^\top F \in \mathbb{R}^{d\times d}$$6 is batch size.

2.2 Gram Anchoring in SSL (DINOv3)

The DINOv3 formulation anchors the patch-relational Gram structure of student features to that of a reference teacher:

• For each image, with $$G = \frac{1}{N} F^\top F \in \mathbb{R}^{d\times d}$$7 patches and feature dimension $$G = \frac{1}{N} F^\top F \in \mathbb{R}^{d\times d}$$8:
• $$G = \frac{1}{N} F^\top F \in \mathbb{R}^{d\times d}$$9: student L2-row-normalized features.
• $g_{\text{diag}} = \text{diag}(G)\in \mathbb{R}^d$0: Gram-teacher L2-row-normalized features.
• Compute Gram matrices: $g_{\text{diag}} = \text{diag}(G)\in \mathbb{R}^d$1 and $g_{\text{diag}} = \text{diag}(G)\in \mathbb{R}^d$2.
• Gram Anchoring Loss is the squared Frobenius norm:

$g_{\text{diag}} = \text{diag}(G)\in \mathbb{R}^d$3

• Only the student is updated; the teacher is either a moving average or periodically reset “Gram teacher”.

3. Integration into Model Objectives

3.1 Prompt Learning (GAPL)

The GAPL model is constructed with three parallel prediction streams:

• Global-invariant (standard global CLIP prediction).
• Gram-anchored (second-order, as above).
• Contextual-anchored.

Each stream yields a cross-entropy term; these are summed:

$g_{\text{diag}} = \text{diag}(G)\in \mathbb{R}^d$4

The Gram Anchoring Loss is $g_{\text{diag}} = \text{diag}(G)\in \mathbb{R}^d$5, a branch-wise cross-entropy.

3.2 Self-Supervised Vision Transformers (DINOv3)

Gram Anchoring is introduced in the late-phase “refinement” stage:

$g_{\text{diag}} = \text{diag}(G)\in \mathbb{R}^d$6

Gram Anchoring is applied to global crops only, starting at high iteration count, with the Gram-teacher updated every $g_{\text{diag}} = \text{diag}(G)\in \mathbb{R}^d$7 steps.

4. Implementation Details and Operational Considerations

• Normalization and scaling: Gram matrices in both settings are normalized: channel-wise mean for GAPL, L2 norm for DINOv3. $g_{\text{diag}} = \text{diag}(G)\in \mathbb{R}^d$8 avoids log singularity. Sigmoid bounds gating in GAPL within $g_{\text{diag}} = \text{diag}(G)\in \mathbb{R}^d$9.
• Teacher management (DINOv3): Gram teacher is a slowly updated EMA; “high-res” teacher can be used by upsampling input before Gram computation, leading to smoother style conditioning.
• Computational cost: Gram matrices are small (e.g., $u = \log(g_{\text{diag}}+\epsilon)$0) and negligible compared to backbone computation; only student gradients are propagated in SSL.
• Branch structure: GAPL runs Gram-anchored alongside other prediction heads; DINOv3 regularizes representation in parallel with other SSL losses.
• Data: GAPL computes Gram statistics per image over patch features; DINOv3 applies Gram loss on global crops exclusively.

5. Empirical Impact

Ablations and benchmarking across both frameworks demonstrate the capacity of Gram Anchoring Loss to improve stability and performance under challenging settings.

GAPL Prompt Learning (Chen et al., 5 Apr 2026)

• Addition of Gram-anchored stream raises average out-of-distribution (OOD) accuracy from 58.77% (global only) to 59.81% (global + Gram) and yields a +3.15 pt boost on the hardest split (ImageNet-A: 47.10% to 50.25%).
• Full combination with context anchoring achieves maximal accuracy (61.12%).

DINOv3 (Siméoni et al., 13 Aug 2025)

• Without Gram Anchoring, dense feature quality degrades late in SSL (VOC mIoU drops from 50.3 to <45 at 1M iters); Gram Anchoring restores it to 53.6 within 10k steps, further improved to 55.7 with high-res Gram.
• 3D keypoint recall improves to 64.4% (vs. 55.0% without Gram Anchoring).
• Depth estimation RMSE drops from 0.307 to 0.285 (to 0.281 with high-res Gram).
• Cosine similarity structure recovers from noisy to well-localized under Gram Anchoring.
• Gram Anchoring has negligible overhead and can be adopted on top of standard objectives.

6. Conceptual Context and Relation to Broader Work

Gram Anchoring Loss is emblematic of increasing reliance on higher-order statistics and relational constraints in modern self-supervised and prompt-learning frameworks. In contrast to first-order objectives (direct alignment of tokens or features), Gram-based losses enforce preservation of feature covariance or structure at a coarser, more robust scale. This structural regularization advances domain robustness (VLMs) and dense prediction quality (SSL ViTs), and can be deployed with minimal alteration to existing pipelines.

A plausible implication is that second-order regularizers like Gram Anchoring will become a standard component wherever patch-level consistency or style robustness is required in transformer architectures for vision. Comparable strategies, such as those leveraging global or covariance statistics, are likely to become increasingly important as model and data scales increase. However, current empirical results show that Gram Anchoring alone is not a universal solution but must be paired with context-specific model objectives to attain full benefit (Chen et al., 5 Apr 2026, Siméoni et al., 13 Aug 2025).