RADIOv2.5 Vision Encoder

• The model integrates multi-teacher distillation with PHI-S normalization and token compression, achieving significant gains in dense prediction and vision-language modeling.
• RADIOv2.5 leverages a Vision Transformer with Cropped Position Embeddings to support arbitrary resolutions, employing multi-resolution and mosaic augmentation for robust performance.
• Empirical results show that RADIOv2.5 consistently outperforms previous baselines on benchmarks like ADE20k and Pascal VOC, demonstrating its practical impact on both perception and VLM tasks.

RADIOv2.5 is an agglomerative vision foundation model that synthesizes advances in multi-teacher distillation, multi-resolution augmentation, and token compression to provide state-of-the-art visual encodings for dense prediction and vision–language modeling. The core architecture is built on a Vision Transformer backbone with Cropped Position Embeddings, designed for robust multi-resolution performance, and is distilled from a diverse teacher set. Substantial empirical performance improvements are demonstrated across dense and VLM benchmarks, with careful ablations elucidating the impact of each architectural and algorithmic innovation (Heinrich et al., 2024).

1. Model Architecture and Tokenization

The backbone of RADIOv2.5 is a Vision Transformer (ViT) with Cropped Position Embeddings (CPE), enabling native support for arbitrary input resolutions. Images are divided into non-overlapping $16\times16$ patches and processed by standard ViT transformer blocks, with CPEs ensuring consistent positional encoding regardless of image size.

Model scale variants include:

• RADIOv2.5-B: ViT-B/16, $\sim$98M parameters
• RADIOv2.5-L: ViT-L/16, $\sim$320M parameters
• RADIOv2.5-H: ViT-H/16, $\sim$553M parameters
• RADIOv2.5-g: ViT-g/14, $\sim$1.1B parameters

Output tokenization partitions the representation as $z = [z_s, z_p^{(1)},\ldots,z_p^{(N)}]$ (Equation 1), where $z_s\in\mathbb{R}^d$ is a single summary token and $z_p^{(i)}\in\mathbb{R}^d$ are per-patch tokens for each $16\times16$ region.

2. Multi-Teacher Distillation Framework

RADIOv2.5 distills knowledge from four teachers: DFN-CLIP, SigLIP, DINOv2-g-reg, and SAM-H, replacing OpenAI-CLIP with SigLIP for improved performance. For each teacher $t$, student features are projected into the teacher space via two MLP adapter heads $\sim$0. The student–teacher feature pairs $\sim$1 and $\sim$2 receive distinct loss functions: cosine similarity for summary tokens, mean squared error (MSE) for patch tokens (Equations 3–7).

Teacher feature distributions are standardized via PHI-S normalization—channel-wise feature rotation and scaling to unit variance and isotropy—allowing uniform weighting across teachers' losses and mitigating teacher-output imbalance and artifacts. Per-teacher "fidelity" (Equation 12) is used to assess learning quality.

3. Multi-Resolution Training and Mosaic Augmentation

To ensure robust performance at different image scales, RADIOv2.5 employs three-stage multi-resolution distillation:

• Stage 1: $\sim$3k iterations at $\sim$4 input resolution
• Stage 2: $\sim$5k at $\sim$6
• Stage 3: $\sim$7k jointly at $\sim$8 and $\sim$9

During training, teacher–student feature matching is performed at the corresponding resolutions. To align with SAM's $\sim$0 input and reduce computation, the mosaic augmentation strategy is used: for inputs smaller than $\sim$1, $\sim$2 smaller images are tiled into a mosaic, processed by SAM, and the output is split back into $\sim$3 submaps (Sec 3.3, Fig. 6). This approach accelerates SAM distillation (by $\sim$4 at low resolutions) and introduces positional diversity, mitigating windowing artifacts.

Mosaic Creation Pseudocode:

$\sim$1

4. Token Compression Strategy

To conform to fixed token budgets in downstream VLMs, RADIOv2.5 integrates Token Merging (ToMe) at the encoder output. Patch tokens are partitioned into "targets" and "sources" via a strided grid, and highest-affinity source tokens are merged into their corresponding targets, discarding merged sources. Affinity is computed using values (not keys) for minimal reconstruction error. Notably, ToMe merging is performed once at the ViT output, yielding a $\sim$5 VLM-score improvement over incremental merging. At $\sim$6 compression (e.g., from $\sim$7 to $\sim$8 tokens), this technique improves VLM accuracy by approximately 6 points over baseline unshuffling (Table 6).

5. Loss Balancing and Training Protocol

Teacher output features are standardized using PHI-S; this harmonization permits uniform loss weighting $\sim$9, addressing imbalance among teacher loss scales. The number of concurrent training partitions modulates performance: fewer partitions enhance summarization (zero-shot, $\sim$0NN) at the cost of memory overhead, while more partitions reduce memory requirements slightly degrading quality. Strategic teacher selection, notably replacing OpenAI-CLIP with SigLIP, yields a $\sim$1-point gain in VLM performance (Sec 3.8, Table 1).

Key training hyperparameters:

• ImageNet-1k pretraining
• DataComp-1B dataset for distillation
• AdamW optimizer ($\sim$2, weight decay $\sim$3)
• 900k total iterations across 3 stages
• Stage 3: two concurrent partitions ($\sim$4 at $\sim$5, $\sim$6 at $\sim$7)
• CPE-based ViT affords true arbitrary resolution

Training is conducted on large compute clusters (256–512 NVIDIA A100s), with mosaic augmentation substantially reducing SAM inference time.

6. Empirical Results and Ablation Insights

The ablation sequence in Table 1 demonstrates successive gains from architectural and algorithmic modifications, starting from the RADIOv2.1-L baseline:

• Adding multi-resolution eliminates mode shifts ($\sim$8 mIoU ADE20k, $\sim$9 VLM avg)
• Incorporating SigLIP as a teacher boosts VLM accuracy
• Scaling to ViT-H backbone further improves overall results
• Introducing ToMe for output token compression yields an additional $\sim$0-point improvement in VLM average (Table 6)

Final models (B/L/H/g) uniformly surpass previous agglomerative baselines (SAM-CLIP, Theia, UNIC, UNIT) on dense benchmarks (ADE20k, Pascal VOC, NYUDv2, Pascal Context; Table 5). In the VILA VLM setup (fixed 196 tokens), RADIOv2.5 encoders (especially H and g) outperform SigLIP and RADIOv2.1-H by 1.5–2.5 points on tasks such as TextVQA, ChartQA, DocVQA, InfoVQA, and OCRBench (Table 8).

7. Inference and Integration

During inference, the pipeline consists of patch embedding, positional encoding via CPE, ViT backbone processing, and optional ToMe-based token compression. The output token sequence—single summary token plus compressed patch tokens—is suitable for direct input to LLMs or task-specific heads.

Inference Pseudocode:

$\sim$2

A plausible implication is that the RADIOv2.5 design—combining ViT+CPE backbone, PHI-S–enabled multi-teacher distillation, staged multi-resolution and mosaic augmentation, and highly effective output token compression—substantially improves the deployability and accuracy of agglomerative vision encoders for both perception-centric and vision–language tasks (Heinrich et al., 2024).