---
title: 'DINOv3: Self-Distillation without Labels'
url: https://www.emergentmind.com/topics/self-distillation-with-no-labels-v3-dinov3
type: topic
---

# DINOv3: Self-Distillation without Labels

Self-Distillation with No Labels v3 (DINOv3) is a self-supervised vision foundation model designed to learn high-quality visual representations from massive image datasets without manual annotation or task-specific adaptation. DINOv3 integrates comprehensive scaling strategies, a novel Gram anchoring method for feature regularization, and post-hoc model tuning protocols, resulting in state-of-the-art global, dense, and multimodal performance across a variety of vision tasks, often without any fine-tuning [2508.10104].

## 1. Self-Distillation Training Paradigm

DINOv3 is built on a teacher–student self-distillation framework, extending the protocol introduced in DINOv2. In each training iteration, a single image yields two global crops (256×256) and eight local crops (112×112); all ten are processed by the student. The objective aggregates three components:
- The **DINO loss** $\mathcal{L}_{\rm DINO}$ applies a SwAV-style Sinkhorn clustering to the class tokens across global crops, matching prototype assignments between student and teacher outputs.
- The **iBOT reconstruction loss** $\mathcal{L}_{\rm iBOT}$ is computed at the patch level on masked local crops, aligning student predictions to normalized teacher patch features.
- The **Koleo regularizer** $\mathcal{L}_{\rm Koleo}$ distributes class-token embeddings uniformly over the hypersphere.

The combined objective is
$$\mathcal{L}_{\rm Pre} = \mathcal{L}_{\rm DINO} + \mathcal{L}_{\rm iBOT} + 0.1\,\mathcal{L}_{\rm Koleo}\,.$$
The teacher network parameters are maintained as an exponential moving average (EMA) of the student weights and are fixed for loss computations.

## 2. Scaling Datasets and Model Architectures

### Data Curation

DINOv3 leverages over 17 billion content-moderated Instagram images. For training efficiency and diversity, several curated pools are sampled:
- **Raw Pool:** Unmodified set.
- **Clustering-Balanced ('LVD-1689M'):** Hierarchical k-means on DINOv2 embeddings, sampling from 25,000 clusters to yield 1,689M balanced images.
- **Retrieval Subset:** Images nearest to seed datasets (ImageNet, etc.) in embedding space.
- **Supervised Pool:** ImageNet-1k, ImageNet-22k, Mapillary samples.

During training, 10% of batches are homogeneous (ImageNet-1k) and 90% are heterogeneous draws. Ablation demonstrates mixed curation achieves optimal results in global and retrieval metrics.

### Model Architecture

The backbone utilizes a custom Vision Transformer (ViT-7B): 40 transformer blocks, 4096 embedding dimension, patch size 16, SwiGLU feed-forward with 8192 hidden dim, 32 heads of dim 128. Rotary positional embeddings (RoPE) with random box jitter in $[-s,s]$, $s\in[0.5,2]$, enhance aspect-ratio resilience. Four register tokens are prepended for background outlier absorption.

### Optimization Protocol

Training proceeds with batch size 4096 across 256 GPUs, constant learning rate (4e-4), weight decay (0.04), and EMA momentum (0.999) following a 100k step warmup. AdamW optimizer in multi-crop mode over 1M iterations exposes the model to 2.56M unique images.

## 3. Gram Anchoring: Spatial Feature Regularization

### Degradation of Dense Features

Extended self-distillation training without regularization degrades the utility of dense representations (e.g., segmentation, depth estimation)—linear probe accuracy may improve, yet patch-wise feature maps lose localization and structure.

### Gram Anchoring Method

Gram anchoring introduces an additional objective to preserve patch-patch similarity:
Given L2-normalized student patch matrix $\mathbf{X}_S$ (global crop) and a Gram teacher $\mathbf{X}_G$ (early EMA snapshot),
$$\mathcal{L}_{\rm Gram} = \bigl\|\,\mathbf{X}_S\,\mathbf{X}_S^\top - \mathbf{X}_G\,\mathbf{X}_G^\top\bigr\|_F^2.$$
This aligns similarity maps rather than absolute feature positions.

### Integration and High-Resolution Teacher Features

After 1M pre-training iterations, a refinement phase incorporates $\mathcal{L}_{\rm Gram}$ (weight 2), updating the Gram teacher every 10k steps. Model performance on dense tasks markedly improves (ADE20k +2 mIoU, better depth RMSE) with no detriment to global metrics. Feeding a twofold higher-resolution image to the teacher, followed by bicubic downsampling, further enhances dense feature quality (+2 mIoU on ADE20k).

## 4. Post-Training Adaptation Protocols

### High-Resolution Feature Adaptation

Fine-tuning for 10k steps on mixed global crop sizes $\{512,768\}$ and local crops $\{112,168,224,336\}$ with continued Gram anchoring ensures the backbone generalizes from standard (224) up to very high resolutions (4096), producing sharp feature maps and steadily improving mIoU.

### Model Distillation

A frozen ViT-7B backbone is used to distill representations into a suite of smaller models (ViT-S, ViT-B, ViT-L, ViT-H+, ConvNeXt variants). Loss computation mirrors pre-training but omits EMA; an efficient multi-student distillation setup allows for teacher-cost sharing and batch loss parallelization.

### Vision–Text Alignment

Text alignment applies a contrastive image–text loss (LiT protocol) with a frozen vision encoder and a new text encoder. Two transformer layers are appended on the vision side; mean-pooled patch and CLS token representations are concatenated. This enables zero-shot global and local retrieval akin to CLIP.

## 5. Experimental Results and Comparisons

### Dense Representation Quality

DINOv3 surpasses previous models (DINOv2, SigLIP2, PEspatial) in feature quality (PCA visualization). Dense linear probes yield ADE20k 55.9 mIoU (+6 over DINOv2), Cityscapes 81.1, VOC 86.6, and best depth RMSE on NYU 0.309, KITTI 2.346. Keypoint matching and unsupervised object discovery set SOTA on NAVI, SPair, and VOC/COCO datasets. Video instance tracking (DAVIS J&F 83.3, +6.7 over DINOv2) and video classification (UCF101 93.5) are on par with leading models.

### Global Representations

Results include 88.4% ImageNet top-1, 81.4% V2, 90.4% ReaL, with best-in-class out-of-distribution robustness (ImageNet-R 91.1, Sketch 71.3, Adversarial 86.9, ObjectNet 79.0, ImageNet-C mCE 19.6).

### System-Level Downstream Tasks

Tables summarize frozen backbone performance on core benchmarks:

| Task                 | Dataset         | Metric         |
|----------------------|----------------|---------------|
| Detection            | COCO           | 65.6 mAP      |
| Segmentation         | ADE20k         | 63.0 mIoU     |
| Depth Estimation     | NYU            | 4.3 %Rel      |
| 3D Pose/Matching     | Re10K/ScanNet  | 86.3 %/35.2 % |

SOTA is observed in multiple segmentation, detection, and depth metrics without end-to-end fine-tuning.

## 6. Ablation Studies and Technical Analyses

Systematic ablation reveals:
- Mixed data curation is optimal for global, linear, and kNN retrieval tasks.
- Layerwise analysis shows global features peak at the final block, while geometric tasks peak near block 32.
- Register tokens eliminate high-norm patch outliers without impacting accuracy, outperforming attn-bias and value gating techniques.
- Early Gram teacher snapshots (100–200k iters) outperform late snapshots for dense feature regularization.
- Twofold downsampling of high-res teacher features maximizes dense task gains.

## 7. Implementation, Computational Considerations, and Domain Adaptation

### Training Infrastructure and Efficiency

Pre-training employs 4096 batch size, AdamW optimizer ($\beta=(0.9,0.999)$), bfloat16, 8-bit matmuls, constant LR (4e-4), WD (0.04), and 1M iterations plus 100k warmup. Gram anchoring uses weight=2, teacher update every 10k, max 3 updates. Multi-student distillation utilizes NCCL all-gather for aggregation and parallel backpropagation.

### Resource and Environmental Impact

ViT-7B pre-training on H100 GPUs accounts for 61k GPU hours (47 MWh, ≈18 t CO₂eq). All distillation and downstream activities total ~9M GPU-hours (≈2600 t CO₂eq).

### Domain Transferability

To adapt DINOv3 to new domains, swap in the target unlabeled corpus, adjust normalization, and maintain core hyperparameters. For metric-based tasks, Gram anchoring or lightweight decoder fine-tuning on synthetic data is recommended. Performance in remote sensing (e.g., on 493M Maxar satellite images) demonstrates state-of-the-art results for canopy height and semantic segmentation using the identical protocol.

---

DINOv3 establishes a unified, scalable, and robust self-supervised paradigm—combining teacher–student distillation, the Gram anchoring regularizer, and flexible post-training adaptation. It yields frozen vision backbones (up to 7B parameters) providing state-of-the-art performance for dense, global, multimodal, and domain-specific applications without end-to-end fine-tuning [2508.10104].

Source: https://www.emergentmind.com/topics/self-distillation-with-no-labels-v3-dinov3