2D Vision Foundation Models

• 2D Vision Foundation Models are large-scale pre-trained models that learn transferable image representations useful for tasks like classification, segmentation, and generation.
• They integrate discriminative and generative approaches through masked, contrastive, and hybrid objectives to optimize semantic alignment and reconstruction.
• VFMs serve as latent interfaces that bridge traditional 2D vision with advanced applications in robotics, 3D modeling, and world state prediction.

2D Vision Foundation Models (VFMs) are large-scale pre-trained models for image and image-like data that learn transferable visual representations and can be adapted to a wide range of downstream tasks with limited task-specific tuning. In the contemporary literature, they are typically characterized by scale of pretraining, generality of transfer, and the ability to support zero-shot, few-shot, prompt-based, or efficient fine-tuned use across recognition, dense prediction, generation, and multimodal systems (Liu et al., 2023). The category now spans discriminative encoders, promptable segmenters, vision-LLMs, generative tokenizers, and task-specialized adaptations, while the boundary between discriminative and generative paradigms has become increasingly porous (Liu et al., 2023).

1. Taxonomy and conceptual scope

The dominant high-level taxonomy separates 2D VFMs into discriminative and generative families. Discriminative VFMs learn decision-oriented or alignment-oriented representations for tasks such as classification, segmentation, detection, retrieval, and dense prediction. Generative VFMs model the visual data distribution and support synthesis, editing, compression, or latent prediction. The survey literature formalizes the generative side by the image distribution $p(\mathbf{x})$, with canonical instantiations including latent-variable models, adversarial models, autoregressive models, and diffusion models (Liu et al., 2023).

This bifurcation is historically useful but increasingly incomplete. Promptable systems such as SAM and SEEM are discriminative in usage yet architecturally generalist; CLIP-style models are trained discriminatively but expose a joint representation space that can guide generation; diffusion and autoregressive models are now being reused for segmentation, classification, compression, and world modeling (Liu et al., 2023). A plausible implication is that “foundation” in vision is less about a single objective family than about a reusable latent interface.

The term itself is not applied uniformly across domains. In radiology, a scoping review notes that there is no universally accepted definition of a VFM, and argues that foundation status should not be claimed on scale alone; it should also reflect heterogeneity, transferability, and broad evaluation (Vergara-Richart et al., 8 Jul 2026). This caution generalizes beyond medical imaging: large backbones or self-supervised pretraining do not automatically imply uniform downstream competence.

2. Pretraining objectives, architectures, and representational bias

The current 2D VFM ecosystem is shaped primarily by three objective families: masked or reconstruction-style learning, contrastive or language-aligned learning, and multi-stage hybrids. In radiology-specific VFMs, 48 studies (71.6%) used self-supervised learning, 19 studies used masking-based SSL, 6 studies used contrastive SSL, and 9 studies (18.8%) used multi-stage or composite SSL pipelines; transformer-based backbones appeared in 53 studies (79.1%) (Vergara-Richart et al., 8 Jul 2026). These figures are domain-specific, but they reflect a broader architectural convergence around ViT-like encoders and SSL.

The pretraining objective materially conditions transfer. In prostate MRI, a reconstruction-oriented MAE-based model and a contrastive DINOv2-style model exhibited different transfer preferences: reconstruction-aligned pretraining favored distortion correction and super-resolution, whereas semantic contrastive pretraining favored classification and segmentation. The paper quantified this with MMD-based representational distances and reported a strong negative correlation between representational distance and performance gain, with Pearson’s $r < -0.8$, $p < 0.05$, supporting the view that downstream utility depends on task alignment rather than generic pretraining strength alone (Huang et al., 22 Jan 2026).

A closely related result appears in image tokenization. VFMTok argues that the most effective tokenizer backbones are VFMs trained with global contrastive learning plus latent masked image modeling, and identifies DINOv2- and SigLIP2-style representations as especially suitable for tokenization and generation (Zheng et al., 18 May 2026). This suggests that pretraining objectives induce not only semantic bias but also a latent geometry that governs compressibility, token efficiency, and generative tractability.

A common misconception is that more pretraining or larger scale should monotonically improve all downstream tasks. The transfer-limit evidence contradicts this: some tasks remain only weakly aligned with a given objective family, so fine-tuning may yield only marginal gains or require large representational shifts (Huang et al., 22 Jan 2026). The practical consequence is that objective design remains central even in the foundation-model regime.

3. Transfer within 2D vision pipelines

Within 2D vision proper, VFMs act as transferable encoders, distillation teachers, and architectural simplifiers. In unsupervised domain adaptation for semantic segmentation, replacing a classical UDA backbone with DINOv2 yielded a system that was both simpler and stronger: with comparable model size, the VFM-UDA method achieved an 8.4× inference speedup over the prior non-VFM state of the art while improving performance by +1.2 mIoU in UDA and +6.1 mIoU in out-of-distribution generalization; with a larger VFM, it still maintained a 3.3× speedup while improving UDA by +3.1 mIoU and OOD generalization by +10.3 mIoU (Englert et al., 2024). The crucial observation was that some classical UDA components, notably FD and HRDA, became redundant once a strong VFM encoder was introduced.

Knowledge transfer to smaller deployable models is another major design pattern. A task-oriented teacher–student pipeline that first adapts the VFM to the target task, then distills task-specific behavior into a small model, and finally fine-tunes the student, outperformed task-agnostic VFM distillation, CLIP pretraining, ImageNet pretraining, and DINO pretraining by up to 11.6%, 22.1%, 13.7%, and 29.8%, respectively, while reducing pretraining compute cost by up to 9×, 4×, and 15× (Vemulapalli et al., 2023). This establishes that VFM transfer is often most effective when the teacher is first made task-aware.

Model-driven VFM construction extends this logic further. KPU unifies CLIP, GroundingDINO, and DINOv2 in a shared latent space, uses a reconstruction regularizer to preserve teacher information, and freezes DINOv2 as a sentinel knowledge base augmented by an adapter. Trained on approximately 15 million images, about 10% of the data used by DINOv2 and Florence-2, it reportedly surpasses DINOv2 by 3.3% on downstream instance segmentation and Florence-2 by 41.5% on zero-shot object detection (Huang et al., 20 Aug 2025). This line of work suggests that large new datasets are not the only route to stronger 2D VFMs; inheritance and unification of existing teachers can also produce general-purpose models.

4. 2D VFMs as interfaces to 3D, robotics, and world modeling

A defining feature of recent work is the use of 2D VFMs as semantic teachers or latent interfaces for non-2D problems. In automotive LiDAR, Seal distills SAM, X-Decoder, OpenSeeD, and SEEM into a point-cloud encoder through cross-modal contrastive learning and temporal regularization, reaching 45.0% mIoU on nuScenes after linear probing, which is 36.9% mIoU above random initialization and 6.1% mIoU above prior art (Liu et al., 2023). DITR similarly projects frozen DINOv2 patch features into 3D point-cloud segmentation and reports state-of-the-art results across indoor and outdoor benchmarks, while D-DITR distills those 2D features into a pure 3D backbone for image-free inference (Zeid et al., 24 Mar 2025).

For occupancy prediction from monocular driving video, zero-shot 2D semantics from CLIP/LSeg and relative depth from Depth Anything can be lifted into 3D after self-supervised scale-and-offset calibration. On nuScenes, this approach surpasses the prior state of the art by 3.34% mIoU for voxel occupancy prediction (Lin et al., 10 Mar 2025). The broader significance is that 2D VFMs can supply semantically rich supervision even when the final target lives in voxel or point space.

In robot learning, Theia distills multiple off-the-shelf VFMs into a compact student optimized for spatial-token representations. The best teacher combination was CDiV—CLIP, DINOv2, and ViT—and Theia-B reached 79.79 ± 0.14 on the MuJoCo subset of CortexBench while also improving real-world manipulation performance (Shang et al., 2024). The associated analysis proposed that higher entropy in feature-norm distributions correlates with better robot-learning utility, with $R = 0.943$ for regular models and $R = 0.638$ for distilled models (Shang et al., 2024). This suggests that not all strong 2D representations are equally useful for embodied control; spatial diversity matters.

World modeling extends the same idea into temporal prediction. VFMF uses a pretrained VFM encoder such as DINOv2 to represent each frame as a dense feature map, compresses those features with a VAE, and then performs autoregressive flow matching in latent space. The resulting latent forecasts can be decoded into semantic segmentation, depth, surface normals, and RGB, and outperform deterministic regression across all tested context lengths on Cityscapes and Kubric MOVi-A (Boduljak et al., 12 Dec 2025). Here the VFM is not just a feature extractor but a world-state representation.

5. Evaluation, robustness, and operational diagnostics

As 2D VFMs have become more heterogeneous, evaluation has shifted from single aggregate scores toward capability-specific diagnostics. AVA-Bench explicitly decomposes visual competence into 14 Atomic Visual Abilities, is built from 26 datasets, and contains 218K image-question pairs. It showed that a 0.5B evaluator LLM can preserve relative VFM rankings obtained with a 7B evaluator while reducing evaluation cost by 8× GPU hours (Mai et al., 10 Jun 2025). Its results emphasize “ability fingerprints”: SigLIP-2 had the best average rank at 2.4, but different models dominated different abilities, with DINOv2 strong on orientation, SAM strongest on color, and language-supervised models strongest overall (Mai et al., 10 Jun 2025).

Robustness evaluation is similarly broadening. A dedicated robustness review argues that VFM assessment is too narrow when limited to a small set of $\ell_p$-bounded attacks and instead must cover distributional shifts, noisy and spatially distorted inputs, and adversarial robustness, including white-box, black-box, poisoning, evasion, and physical adversarial patches (Gupta et al., 22 Aug 2025). The paper formalizes the ideal invariance criterion as

$$f(x') = f(x), \quad \text{s.t. } \|x' - x\|_p \le \epsilon,$$

but stresses that real deployment requires metrics such as corruption error, fooling rate, retention rate, and certified bounds, not clean accuracy alone (Gupta et al., 22 Aug 2025).

In autonomous driving, VFM features combined with density estimation offer a model-agnostic input monitor for both semantic and covariate shift. The strongest combination in the benchmark was Grounding DINO Swin + Normalizing Flow, with representative FPR95 values of 0.16 on Lost and Found, 0.02 on Bravo-Synobj, 0.00 on Foggy Cityscapes $(i=0.2)$, and 0.02 on Bravo-Synrain (Keser et al., 14 Jan 2025). The same monitor preferentially filtered high-risk samples, improving downstream segmentation mIoU on IDD, which indicates that density in VFM feature space can function as a proxy for operational risk (Keser et al., 14 Jan 2025).

These results challenge another common misconception: broad zero-shot or benchmark success does not imply uniform reliability. Capability can be anisotropic across abilities, shifts, and deployment conditions, and diagnostic evaluation is necessary to localize failure modes.

6. Generative uses, domain-specific deployments, and unresolved questions

A major recent development is the reuse of 2D VFMs as tokenizers, codecs, and semantic priors for noncanonical inputs. VFMTok builds a generalist tokenizer on top of a frozen VFM encoder using region-adaptive quantization and a semantic reconstruction objective. In discrete autoregressive generation it reports 3 times faster convergence and a state-of-the-art gFID of 1.36 on ImageNet class-conditional synthesis; in continuous-space generation it reports gFID of 1.25; and the tokenizer reduces length from 576 tokens to 256 tokens, yielding about 4× generation speedup while enabling high-fidelity class-conditional synthesis w/o CFG (Zheng et al., 18 May 2026). The underlying claim is that frozen VFMs can serve as semantically rich tokenizers rather than only as encoders.

Autoregressive VFM compression pushes the generative reuse further. By repurposing next-token prediction for entropy coding, pre-trained VFMs become image codecs with bitrate

$R = \sum_{i=1}^{N} -\log_2 p(z_i \mid z_{<i}),$

and on Kodak and CLIC2020 they achieve extremely low bitrates, below 0.1 bpp in some cases, while sometimes exceeding specialized codecs in perceptual quality (Phung et al., 5 Sep 2025). The principal limitation is complexity: sequential autoregressive decoding remains substantially slower than conventional learned codecs (Phung et al., 5 Sep 2025).

Domain-specific deployments show that 2D VFMs are useful far beyond natural images. In microscopy, frozen VFM embeddings consistently improved both interactive pixel classification and object classification over hand-crafted features; domain-specific VFMs were strongest for random-forest pipelines, SAM2 was best for attentive probing, and DINOv3 was consistently the weakest among the evaluated VFMs (Teuber et al., 20 Mar 2026). In Parkinson’s disease diagnosis from limited 3D NM-MRI and QSM, MRN used slice-wise DINOv2 encoders, ROI-wise fusion, auxiliary segmentation, and supervised contrastive learning to reach 86.0% challenge accuracy on only 300 labeled training/validation scans, outperforming the second-place method by 5.5% (Shaodong et al., 22 Sep 2025). In ECG-based stress recognition, SONIC converted 1D signals into STFT spectrograms and fused multiple 2D backbones—ViT, EfficientNet, ResNet50, and VGG19—reporting 99.36% accuracy for binary stress recognition and 97.66% for three-class classification on WESAD (Phukan et al., 2024).

The open questions are now less about whether 2D VFMs transfer at all and more about how, when, and at what cost. The literature identifies at least four persistent tensions: pretraining objective versus downstream task structure (Huang et al., 22 Jan 2026); broad benchmark strength versus ability-specific weakness (Mai et al., 10 Jun 2025); perceptual or semantic quality versus decoding complexity in generative applications (Phung et al., 5 Sep 2025); and claimed foundation status versus incomplete heterogeneity, reproducibility, or deployment validation (Vergara-Richart et al., 8 Jul 2026). Taken together, these results suggest that the mature view of 2D VFMs is not that they are universally optimal image models, but that they are increasingly general latent infrastructures whose utility depends on alignment, diagnostics, and the design of transfer mechanisms.