MedDINOv3: ViT-Based Medical Segmentation

• MedDINOv3 is a framework that adapts DINOv3-powered Vision Transformers for medical image segmentation, addressing dense prediction challenges.
• It employs multi-scale token aggregation and a lightweight decoder to achieve precise organ and tumor delineation on high-resolution CT and MRI scans.
• The method uses domain-adaptive self-supervised pretraining on the CT-3M dataset to improve generalizability and state-of-the-art performance across benchmarks.

MedDINOv3 is a framework for adapting vision foundation models, specifically DINOv3-powered Vision Transformers, to medical image segmentation tasks such as organ and tumor delineation in CT and MRI scans. Developed to address challenges in generalizability and dense prediction quality, MedDINOv3 leverages architectural refinements and a domain-adaptive self-supervised pretraining regimen on the CT-3M dataset (3.87M axial CT slices), yielding state-of-the-art performance across heterogeneous medical segmentation benchmarks. The approach demonstrates how large-scale, natural-image-pretrained models can be made suitable for medical semantic segmentation through multi-scale token aggregation and tailored representation learning.

1. Motivation and Core Challenges

MedDINOv3 targets two principal hurdles in adapting vision foundation models to medical segmentation:

• ViT Backbones and Dense Segmentation: Conventional Vision Transformer architectures underperform relative to task-specific CNNs for dense medical segmentation due to faint locality bias and insufficient granular spatial priors.
• Domain Gap: Pretrained representations from natural images struggle with transferability to radiological modalities, given extensive differences in intensity, texture, and semantic context.

These limitations are pronounced in real-world clinical scenarios, where segmentation models must generalize across institutions, imaging protocols, and patient populations. MedDINOv3’s design specifically seeks to overcome these barriers.

2. Architectural Design and Multi-Scale Token Aggregation

MedDINOv3 employs a plain ViT backbone initialized from DINOv3 and augments it with multi-scale token aggregation. Key elements include:

• Multi-Scale Features: Patch tokens from intermediary transformer blocks (typically blocks 2, 5, 8, and 11 in a 12-layer ViT) are aggregated, yielding hierarchical representations crucial for precise boundary definition.
• Lightweight Decoder: The ViT-derived encoder is paired with a decoder architecture inspired by Primus, comprising transposed convolutions, LayerNorm, and GELU activations, to efficiently upsample multi-scale features into full-resolution segmentation outputs.
• High-Resolution Inputs: Training and inference operate at high image resolutions (e.g., 896 × 896), preserving fine anatomical details. Unlike some approaches, MedDINOv3 does not reduce patch size post-pretraining, preserving computational tractability and spatial fidelity.

Figure 1 in the original paper presents PCA projections of feature activations at various stages, illustrating the emergence of dense, spatially coherent representations post-aggregation.

3. Domain-Adaptive Self-Supervised Pretraining on CT-3M

MedDINOv3 introduces a multi-stage recipe for domain adaptation to medical imaging, leveraging CT-3M—the largest curated dataset of axial CT slices spanning 16 public sources.

• Stage 1: DINOv2-style SSL on CT-3M, combining global/local self-distillation losses, patch-wise reconstruction (L_iBOT), and feature diversity regularization (L_Koleo):

$$L_\text{stage1} = L_\text{DINO} + L_\text{iBOT} + 0.1 \times L_\text{Koleo}$$

• Stage 2: Gram Anchoring. To retain rich patch-level spatial structure, MedDINOv3 introduces a Gram matrix loss, aligning the pairwise similarity structure of patch features between the current “student” and a preserved “Gram teacher:”

$$L_\text{Gram} = \left\|\mathbf{X}_S \mathbf{X}_S^\top - \mathbf{X}_G \mathbf{X}_G^\top \right\|_F^2$$

where $\mathbf{X}_S,\mathbf{X}_G$ are $P \times d$ matrices of $\ell_2$-normalized patch features.

• Stage 3: Adaptation to high-resolution crops, extending Gram anchoring to maintain feature quality with larger input sizes.

Figure 2 in the paper visualizes high-resolution cosine similarity matrices, demonstrating that post-adaptive pretraining, MedDINOv3’s features retain fine-grained spatial distinctions essential for segmentation accuracy.

4. Segmentation Performance Across Benchmarks

MedDINOv3 was tested on four public benchmarks spanning organ and tumor segmentation in CT and MRI:

Benchmark	Task	Performance Gain (DSC)	Key Metric
AMOS22	OAR segmentation	+2.57% over nnU-Net	Dice Similarity Coefficient
BTCV	OAR segmentation	+5.49% over nnU-Net	Dice Similarity Coefficient
KiTS23	Kidney tumor segm.	70.68%	Dice Similarity Coefficient
LiTS	Liver tumor segm.	75.28%	Dice Similarity Coefficient

Performance on the tumor segmentation tasks matches or eclipses that of leading specialized architectures. Additional metrics such as normalized surface Dice (NSD) confirm enhanced boundary accuracy. The ablation studies in Table 2 of the paper validate contributions from multi-scale aggregation and Gram anchoring.

5. Methodological Implications and Limitations

MedDINOv3 demonstrates that:

• Unified ViT Backbones: With targeted adaptation, a single ViT backbone can deliver strong segmentation performance across modalities and tasks, streamlining architecture design in clinical AI pipelines.
• Effective Domain Bridging: Domain-specific self-supervised pretraining (especially with Gram anchoring and multi-stage loss structure) substantially narrows the gap between web-scale natural image FMs and radiological applications.
• Architectural Simplicity: Revisiting plain ViTs and focusing on multi-scale token aggregation—as opposed to more intricate modules—yields efficient yet powerful dense predictors.

A plausible implication is that further scaling of pretraining data and backbone size would continue to yield improvements, provided adequate fine-grained adaptation. Limitations highlighted include the current focus on 2D slice-based segmentation and the need for further advances in 3D volumetric adaptation.

6. Future Directions and Research Significance

The outlook for MedDINOv3 centers on:

• Optimization of Pretraining Recipes: Alternative or enhanced domain adaptation strategies, especially in Gram matrix preservation and loss function engineering.
• 3D and Multi-Modality Extension: Extending the framework to fully volumetric segmentation and integrating other modalities (beyond CT/MRI) for broader clinical applicability.
• Scalability and Clinical Translation: Given the method’s demonstrated cross-institution generalization, future research may apply MedDINOv3 in federated, multi-center scenarios for robust real-world segmentation.
• Broader Foundation Model Applications: Findings advocate for the continued adaptation of vision FMs to specialized domains through targeted self-supervised and architectural advances, influencing medical AI beyond segmentation.

In conclusion, MedDINOv3 offers a principled approach for leveraging vision foundation models in medical image segmentation, centered on multi-scale token aggregation, domain-adaptive pretraining, and efficient dense prediction. Its results on diverse clinical benchmarks establish a foundation for further unified, scalable medical imaging solutions (Li et al., 2 Sep 2025).