TIPSv2: Advanced Vision-Language Pretraining

• TIPSv2 is a vision-language pretraining model family that aligns dense image patch representations with text embeddings for tasks like segmentation, retrieval, and classification.
• It leverages innovations such as a refined masked image modeling loss (iBOT++), patch-level distillation, and a head-only EMA mechanism to overcome limitations in dense patch-text alignment.
• Extensive experiments across 20 datasets demonstrate its state-of-the-art performance and efficiency, making it a key advancement in vision-language research.

TIPSv2 is a family of vision-language pretraining (VLP) models designed to advance the alignment between dense image patch representations and text embeddings, enhancing foundational capabilities for downstream computer vision tasks such as classification, retrieval, semantic segmentation, and depth prediction. The core architectural and training innovations in TIPSv2—namely, a refined masked image modeling loss (iBOT++), patch-level distillation, a head-only exponential moving average (EMA) mechanism, and a multi-granularity synthetic caption sampling—address and overcome persistent limitations in dense patch-text alignment, resulting in state-of-the-art performance across a diverse set of evaluation benchmarks (Cao et al., 13 Apr 2026).

1. Model Architecture and Objectives

TIPSv2 utilizes a dual-encoder architecture. The image encoder $f$ is a Vision Transformer (ViT), producing a global CLS embedding ($e^g = f^g(I) \in \mathbb{R}^D$) as well as a set of patch embeddings $\{e_i=f_p(I)_i\}_{i=1}^N$. The text encoder $g$ is a Transformer that encodes a caption $T$ into a single CLS embedding $e^t = g(T) \in \mathbb{R}^D$. To leverage distinct forms of supervision, TIPSv2 incorporates two separate CLS heads on $f^g(I)$: one trained on “web” alt-text and the other on synthetic captions.

The composite pretraining loss is:

$$L = L_{\text{CLIP}} + L_{\text{DINO}} + L_{\text{iBOT++}}$$

• $L_{\text{CLIP}}$ (Contrastive Image–Text Loss, CLIP-style): Implements symmetric cross-entropy between global image and text encodings across a batch, with separate terms for each caption source type ($\beta \in \{ \text{web, synth} \}$). For batch size $e^g = f^g(I) \in \mathbb{R}^D$0 and learnable temperature $e^g = f^g(I) \in \mathbb{R}^D$1,

$e^g = f^g(I) \in \mathbb{R}^D$2

• $e^g = f^g(I) \in \mathbb{R}^D$3 (Global Self-Distillation): For $e^g = f^g(I) \in \mathbb{R}^D$4 local crops in a batch, with a teacher and student network (both sharing the main encoder; see Section 2.2 for EMA),

$e^g = f^g(I) \in \mathbb{R}^D$5

• $e^g = f^g(I) \in \mathbb{R}^D$6 (Enhanced Patch-Text Alignment): Generalizes masked image modeling by supervising all patch tokens (both visible and masked). If $e^g = f^g(I) \in \mathbb{R}^D$7 denotes masked patches ($e^g = f^g(I) \in \mathbb{R}^D$8), $e^g = f^g(I) \in \mathbb{R}^D$9 the unmasked, and for patch $\{e_i=f_p(I)_i\}_{i=1}^N$0, $\{e_i=f_p(I)_i\}_{i=1}^N$1 (student) and $\{e_i=f_p(I)_i\}_{i=1}^N$2 (teacher), the loss is:

$\{e_i=f_p(I)_i\}_{i=1}^N$3

where $\{e_i=f_p(I)_i\}_{i=1}^N$4 is cross-entropy. In practice, $\{e_i=f_p(I)_i\}_{i=1}^N$5, so over all patches,

$\{e_i=f_p(I)_i\}_{i=1}^N$6

2. Distillation and EMA Scheme

TIPSv2 introduces two key modifications to the distillation and Exponential Moving Average (EMA) regime:

• Patch-Level Distillation: The method utilizes a frozen, larger teacher ($\{e_i=f_p(I)_i\}_{i=1}^N$7) exposed to the unmasked image and a randomly initialized student ($\{e_i=f_p(I)_i\}_{i=1}^N$8) observing the masked image. For the student to escape local minima (or “pretrained basin”), it is crucial for the student to be randomly initialized and for the teacher to remain frozen. Transitioning the mask ratio to zero enables effective distillation:

$\{e_i=f_p(I)_i\}_{i=1}^N$9

• Head-Only EMA: Unlike standard DINO/iBOT self-distillation where both encoder and head adopt EMA updates, TIPSv2 applies EMA exclusively to the projection head (since the CLIP-style loss prevents collapse). For projection head $g$0 and decay parameter $g$1:

$g$2

The encoder remains shared ($g$3). This reduces memory usage by roughly 50% in EMA parameters, with negligible accuracy drop.

3. Synthetic Caption Sampling and Dual-CLS Mechanism

TIPSv2 leverages multi-granularity synthetic captions to maximize the downstream utility and prevent overfitting on caption length. Caption sources comprise:

• Web alt-text (noisy, object-centric)
• PaliGemma synthetic (short, spatial)
• Gemini-Flash synthetic (detailed, multi-sentence)

Under the dual-CLS setup:

• CLS$g$4 is supervised by web alt-text.
• CLS$g$5 alternates between PaliGemma and Gemini captions, each batch ($g$6).

This strategy enforces diversity in supervision, avoiding trivial discrimination arising from caption length alone and introducing fine-grained spatial reasoning.

4. Training Regime, Hyperparameters, and Model Scaling

TIPSv2 is pretrained on a 116M pair subset of WebLI with both web and synthetic captions. The pretraining schedule consists of two stages:

1. Stage 1 (Low-Resolution):
   • 90k steps, batch size 8192
   • Global crop 224px, 1 view; local crops 98px, 6 views
2. Stage 2 (High-Resolution Adaptation):
   • 9k steps, batch size 4096
   • Global 448px; local 140px

A 75% masking ratio is applied. The loss is formed as $g$7 with weights $g$8 (DINO), $g$9 (iBOT++), and the optimizer is Adafactor, following the TIPS v1 learning-rate and warmup defaults.

Distillation to smaller models (ViT-L/14, ViT-B/14, SoViT-400M) proceeds for 100k steps, mask ratio zero, and then high-res adaptation as above.

5. Empirical Performance and Ablation

Extensive evaluation was conducted on 9 tasks across 20 datasets, including dense image-text alignment (zero-shot segmentation), global retrieval (COCO, Flickr30k, DOCCI), zero-shot classification (ImageNet-1K), and image-only dense/global tasks (segmentation, depth, fine-grained retrieval).

Key empirical findings include:

Task	Metric	TIPS v1	TIPS v2 (iBOT++)
Zero-Shot Segmentation (ViT-L/14)	PC59 (mIoU)	33.5	37.1
	PC60 (mIoU)	30.4	33.9
	VOC21 (mIoU)	30.5	44.4
	ADE150 (mIoU)	20.8	24.7
COCO Image $T$0 Text (ViT-g)	R@1 (%)	74.0	75.7
COCO Text $T$1 Image (ViT-g)	R@1 (%)	59.4	60.7
DOCCI (long text)	R@1 (%)	58.8	72.8
Flickr30k	R@1 (%)	93.0	95.1
ImageNet zero-shot classification	Top-1 (%)	79.9	80.7

Ablation studies indicate iBOT++ yields strong improvement on zero-shot segmentation (ADE150 mIoU +14.1), while the multi-granularity caption setup further enhances retrieval and dense task performance. The head-only EMA approach preserves nearly all accuracy while reducing memory requirements by approximately 42% on ViT-B.

6. Deployment Recommendations and Domain Adaptation

Guidelines for deploying TIPSv2:

• For maximal dense alignment (segmentation, depth), ViT-g/14 TIPSv2 is most effective.
• For retrieval/classification mixtures, ViT-L/14 balances model cost and performance.
• For computational efficiency, ViT-B/14 and SoViT-400M distilled by the TIPSv2 patch-distill recipe are recommended.
• To adapt to new domains, maintain a 75% mask with the iBOT++ objective, use dual-CLS with multi-granularity captions, and employ head-only EMA when a contrastive (CLIP-style) loss is present.

7. Significance and Research Implications

TIPSv2 addresses a fundamental barrier in VLP—effective dense patch-text alignment—through architectural innovations and targeted modifications to training regimes. The practical improvements in segmentation, retrieval and global classification highlight the efficacy of directly supervising all patch tokens, leveraging diverse caption supervision, and streamlining self-distillation via head-only EMA. The empirical gains and generality suggest viability across a broad array of computer vision-linguistic transfer tasks (Cao et al., 13 Apr 2026). All models, training logs, and code are released for further research and development.