Visual Foundation Model Representations

• Visual Foundation Models are vision neural networks that produce general-purpose visual representations for diverse tasks such as object recognition and pose estimation.
• CLIP uses contrastive image–text training to deliver robust, global semantic descriptors, while DINOv2 preserves dense, spatially detailed features for geometric accuracy.
• Hybrid pipelines combining both semantic and geometric features offer improved performance in applications, balancing language grounding with precise spatial understanding.

A Visual Foundation Model (VFM) is defined as a vision neural network—typically a transformer-based architecture—trained on vast, diverse image collections with self-supervised, supervised, or multimodal (vision–language) objectives. VFMs are designed to produce general-purpose, pre-trained visual representations that can transfer effectively to a wide range of downstream tasks. The characteristics, structure, and practical implications of VFM representations are outlined below, drawing from the most recent comprehensive studies of their properties (Sarowar et al., 8 Dec 2025).

1. Architecture and Embedding Functions

CLIP (Contrastive Language–Image Pretraining)

• Backbone: ViT-B/32 (12 transformer layers, patch size 32×32, hidden dim $d=768$)
• Embedding Function: The output is a 512-dimensional global semantic vector $$z_{\text{CLIP}} = f_{\text{CLIP}}(x) = W_v \cdot \text{ViT}(x)_{\text{[CLS]}} \in \mathbb{R}^{512}$$, where $W_v$ is a learned projection from the [CLS] token.
• Objective: Symmetric contrastive loss over $N$ image–text pairs within a batch:

  $$L_{\text{CLIP}} = - \frac{1}{2N} \sum_{i=1}^N \Bigg[ \log \frac{\exp( \text{sim}(z^{i}_{\text{vis}}, z^{i}_{\text{txt}})/\tau ) } { \sum_{j=1}^N \exp(\text{sim}(z^{i}_{\text{vis}}, z^{j}_{\text{txt}})/\tau) } + \log \frac{ \exp(\text{sim}(z^{i}_{\text{vis}}, z^{i}_{\text{txt}})/\tau) } { \sum_{j=1}^N \exp( \text{sim}(z^{j}_{\text{vis}}, z^{i}_{\text{txt}})/\tau ) } \Bigg]$$

where $\text{sim}(u,v)=u^\top v/\|u\|\,\|v\|$, $\tau$ is learnable.

DINOv2 (Self-Supervised Visual Foundation Model)

• Backbone: ViT-B/14 (12 layers, patch size 14×14, hidden dim $d=768$)
• Embedding Function: Per-image output is a dense grid $$f_{\text{DINOv2}}(x): \mathbb{R}^{H\times W\times 3} \rightarrow \mathbb{R}^{(H/14)\times (W/14) \times 768}$$.
• Objective: Self-distillation with teacher–student setup; patch embedding features are encouraged to be consistent under augmentations.

  $$L_{\text{DINO}} = \sum_{p \in \{ \text{views} \} } D(\text{softmax}( h_s(x_p)/\tau_s ), \text{stopgrad}(\text{softmax}( h_t(x_p)/\tau_t )))$$

  $h_s, h_t$ are student/teacher ViTs; $D$ is e.g. cross-entropy.

2. Semantic vs. Geometric Representation Properties

Global Semantic Embedding (CLIP)

CLIP's training for image-text alignment causes the model to encode a global, holistic, and context-sensitive semantic representation. The output vector $z_{\text{CLIP}}$ is highly sensitive to object identity, context, and semantic affordances but discards much spatial and geometric detail. The [CLS] token pools the entire patch-wise feature map plus position encodings into a compact, invariant descriptor.

Dense Geometric Features (DINOv2)

Self-supervised objectives in DINOv2 preserve spatially distributed, high-fidelity local features. Each patch embedding $F(u,v)$ maintains fine-edge, contour, and geometric cues, with the grid covering the full input spatial extent. Per-patch attention maps provide additional geometric structure—self-similarity, symmetry, and repeated pattern detection—that is robust to moderate viewpoint and photometric changes.

3. Representation Dimensionality and Structure

Model	Input Size (RGB)	Patch Grid	Feature Dim	Output Structure
CLIP	224×224	7×7	512	$\mathbb{R}^{512}$ (global)
DINOv2	224×224	16×16	768	$\mathbb{R}^{16×16×768}$ (dense)

CLIP employs a projection to a low-dimensional global vector, with spatial information collapsed. DINOv2 operates at higher spatial resolution, generating a dense set of patch-wise features. Pooling over these patch features can produce a global representation if required, but spatial arrangement is preserved by default.

4. Quantitative Evaluations and Losses in Downstream Tasks

For 6D object pose estimation (e.g., hand–object grasping), both semantic and geometric losses are applied:

• Geometric Loss ($L_\text{geo}$): Measures $\ell_2$ alignment between predicted and ground-truth rigid transformations over $m$ model points:

  $$L_\text{geo}(\hat R, \hat t) = \frac{1}{m} \sum_{i=1}^m \| \hat R x_i + \hat t - (R_{gt} x_i + t_{gt}) \|_2$$

• Semantic Loss ($L_\text{sem}$, used for CLIP): Cross-entropy on a classifier over the concatenated image and text features.

Empirical benchmarks on the Driller object show:

Metric	CLIP-Based	DINOv2-Based
ADD (mm, ↓)	32.17	28.45
ADD-S (mm, ↓)	32.17	29.12
Rotation Error (°, ↓)	11.68	9.34
Translation Error (mm, ↓)	20.00	17.52

DINOv2 consistently shows better geometric accuracy; CLIP achieves superior semantic alignment (semantic loss not applicable to DINOv2 directly).

5. Representation Selection: Application-Specific Guidance

• CLIP (Semantic Consistency): Preferable where robust object recognition, affordance reasoning, or grounding natural-language instructions to objects is prioritized. This includes ambiguous contexts, visually similar objects requiring disambiguation by scene, or zero-shot generalization to novel categories. A typical tolerance is a ~15–20% increase in ADD for stronger semantic robustness.
• DINOv2 (Geometric Precision): Selected for millimeter-accuracy pose estimation in clutter, occlusions, multi-object scenes, and direct dense correspondence-based localization. DINOv2 features allow for partial occlusion handling and implicit depth encoding from RGB alone.
• Hybrid Pipelines: A practical approach is a two-stage process; global semantic filtering/localization with CLIP, followed by geometric pose refinement with DINOv2 dense features and geometric alignment algorithms (PnP/ICP).

6. Synthesis and Implications for VFM Research

VFMs, as exemplified by CLIP and DINOv2, realize fundamentally different inductive biases in their learned representations due to their pretraining tasks:

• CLIP, via image–text contrastive learning, excels as a global, semantic descriptor for recognition, language grounding, and instruction following, but at a cost to spatial/geometric detail.
• DINOv2, through self-distillation and local patch consistency, produces a geometric–topological feature grid suitable for pose estimation, manipulation, and tasks sensitive to spatial structure.

Representation selection in VFM-based pipelines must be driven by the semantic/geometric demands of the intended application. For tasks spanning both domains, staged or hybrid approaches leveraging the complementary nature of semantic and geometric VFM representations are empirically substantiated as best practice (Sarowar et al., 8 Dec 2025).