Pre-trained Vision Models (PVMs)

• Pre-trained Vision Models (PVMs) are visual encoders pre-trained on large-scale data and adapted for diverse tasks like recognition, grounding, and control.
• They utilize techniques such as patch tokenization, Transformer attention, and contrastive alignment to overcome sample-efficiency bottlenecks and improve generalization.
• PVMs play a key role in both vision-only and multimodal systems, driving advances in robotics, continual learning, and safety-critical applications.

Pre-trained vision models (PVMs) are visual encoders learned before downstream deployment and then transferred, adapted, or probed in later tasks. In contemporary usage, the term covers both vision-only backbones and the visual components of vision-and-language pretrained models, including systems that learn joint or aligned image-text representations and then serve as reusable perception modules for recognition, grounding, retrieval, reasoning, and control (Long et al., 2022, Nguyen et al., 2022, Qi et al., 2024). Across this literature, the central premise is that large-scale pretraining fixes much of the sample-efficiency and generalization bottleneck of task-specific visual learning, but it also shifts attention toward representation geometry, adaptation strategy, data regime, safety, and maintenance.

1. Conceptual scope and architectural families

A standard abstraction in the survey literature decomposes vision-and-language pretrained systems into four stages: raw visual and language input, unimodal representation, cross-modal interaction, and learned multimodal representation (Long et al., 2022). Within that space, PVMs appear both as standalone image encoders and as the visual side of multimodal systems.

Family	Visual representation	Representative systems
Single-stream fusion	Region, grid, or patch tokens concatenated with text	VisualBERT, VL-BERT, UNITER, OSCAR, ViLT
Dual-stream / co-attention	Separate visual and textual streams with later interaction	ViLBERT, LXMERT, ERNIE-ViL
Dual-encoder / shared embedding	Independent image and text encoders aligned in one space	CLIP, ALIGN, LightningDOT, COOKIE

A major historical shift has been from detector-dependent region features toward end-to-end patch tokenization. Early VLPMs commonly used Faster R-CNN region-of-interest features with spatial embeddings, which imposed an object-centric ontology on the model. Later systems increasingly adopted grid, pixel, and then ViT-style patch representations, partly because they remove detector bottlenecks and support scalable Transformer pretraining (Long et al., 2022, Nguyen et al., 2022). This trend also blurs the line between “vision models” and “vision-LLMs”: in CLIP-like systems, the visual encoder is both an image backbone and one side of a cross-modal embedding model.

The visual encoder’s role differs by family. In fusion models it produces tokens suitable for token-level interaction; in dual encoders it produces an independently useful embedding for retrieval and zero-shot classification; in embodied systems it often becomes a frozen perceptual front end whose outputs condition a downstream policy. This suggests that “PVM” denotes not one architecture class but a transfer-learning regime centered on reusable visual representations.

2. Pretraining objectives and representation structure

Transformer attention is the dominant primitive in contemporary PVMs and VLPMs, typically written as

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

(Long et al., 2022). What differentiates model families is less the primitive than the supervision imposed on the resulting representation.

The survey literature identifies four major objective classes. Cross-modal masked language modeling predicts masked text from text plus visual context; masked visual modeling predicts masked regions or patches from surrounding visual and textual evidence; image-text matching performs pair verification; and contrastive alignment learns a shared embedding space in which matched image-text pairs are close and mismatched pairs are separated (Long et al., 2022, Nguyen et al., 2022). In CLIP-like models, the resulting classifier can be synthesized from language, with image embeddings scored against text embeddings by cosine similarity and a learned temperature, rather than by a conventional learned linear head (Miao et al., 2023).

Vision-only pretraining has increasingly moved toward hybrid objectives. HVP-MTL combines supervised multi-label classification, SWaV-style contrastive clustering, masked image modeling, and momentum distillation in a single shared-encoder recipe. Its total objective is

$$\mathcal{L} = \alpha_1 \left(\mathcal{L}_+^{\text{mcls}} + \mathcal{L}_-^{\text{mcls}}\right) + \alpha_2 \mathcal{L}^{\text{cl}} + \alpha_3 \mathcal{L}^{\text{mim}} + \alpha_4 \mathcal{L}^{\text{mom}},$$

with $\alpha_1 = 0.001$, $\alpha_2 = 0.02$, $\alpha_3 = 1.0$, and $\alpha_4 = 10.0$. Using Tencent-ML pretraining, it reports 85.3% top-1 accuracy on ImageNet-1K with Swin-B, 47.9 box AP on COCO object detection for Mask R-CNN, and 50.6 mIoU on ADE20K with UperNet (Qian, 2023). This supports the claim that semantic supervision and self-supervised structure learning are complementary rather than redundant.

A distinct line of work argues that the decisive property for robotics transfer is not merely objective class but the ability to induce object-centric structure from the available data. A data-centric study finds that DINO and iBOT outperform MAE across visuomotor control and perception tasks, but they degrade on non-(single-)object-centric data, and that this degradation is strongly correlated with diminished object-centricity. The reported correlation between objectness and manipulation success is 0.72 (Wen et al., 10 Mar 2025). SlotMIM addresses this by reducing prototype count to create a semantic bottleneck and adding cross-view consistency and slot-level contrastive learning, which suggests that object-centric representation formation is a primary latent variable governing transfer quality on cluttered or scene-centric data.

3. Adaptation and transfer

Because PVMs are usually large, adaptation strategy is often as important as pretraining. In CLIP-style systems, a common baseline is to freeze the backbone and adapt prompts or small adapters. MuDPT argues that uni-modal prompt tuning is incomplete for vision-language pretrained backbones because it breaks the original cross-modal alignment. It therefore inserts deep textual prompts and deep visual prompts across Transformer depth and couples them with an Injection Model that performs hierarchical bi-directional prompt fusion:

$$T', V' = Injection(T, V), \qquad \hat{T}, \hat{V} = T + V', \; V + T'.$$

Only prompts and the Injection Model are optimized; the CLIP backbone remains frozen. On 11 datasets in a 16-shot setting with CLIP ViT-B/16, MuDPT reports average accuracies of 64.65 for zero-shot CLIP, 73.18 for CoOp, 75.07 for CoCoOp, and 81.38 for MuDPT; in base-to-new generalization it reaches 78.39 average versus 75.83 for CoCoOp (Miao et al., 2023). The main implication is that when a “vision model” is actually the visual half of a multimodal encoder, adaptation that preserves multimodal alignment can improve both few-shot recognition and generalization.

In continual class-incremental learning, a related pattern emerges. A CLIP-based method compares prompt tuning, self-attention adapters, and a Linear Adapter inserted after the image encoder, together with a parameter-retention mechanism. The simplest configuration—a single Linear Adapter plus selective retention—produces the best results, reaching 84.2 Avg / 76.9 Last on CIFAR-100 B0-10 and 85.3 Avg / 76.8 Last on ImageNet-100 B0-10 (Liu et al., 2023). This suggests that, for strong frozen PVMs, low-capacity feature-space remapping can be more stable than higher-capacity adaptation.

Continual semantic segmentation exhibits a related but more counterintuitive effect. A study of Direct Fine-Tuning (DFT) with ResNet101 and Swin-B backbones finds that PVMs already retain previous knowledge surprisingly well, and that the major source of apparent forgetting is classifier drift rather than representation collapse. Its DFT* variants freeze the backbone, freeze previous classifiers, and in one case pre-allocate future classifiers; across Pascal VOC 2012 and ADE20K, DFT* is competitive with or better than sixteen state-of-the-art methods while using substantially fewer trainable parameters and less training time (Zhang et al., 6 Aug 2025). A plausible implication is that with sufficiently strong pretrained backbones, continual adaptation is often a problem of preserving feature-classifier alignment rather than relearning visual semantics.

Cross-lingual transfer pushes the same argument into multimodal language space. XVL-MAML treats language-specific image-text alignment as a MAML task and augments adaptation with a bidirectional contrastive loss. Applied to xUNITER and UC2, it improves zero-shot cross-lingual transfer on XVNLI, xGQA, MaRVL, and xFlickr&Co, including gains such as 59.0 to 63.0 on XVNLI for xUNITER and 19.7 to 21.3 IR on xFlickr&Co for UC2 (Hu et al., 2023). This suggests that pretrained multimodal encoders require not only stronger pretraining but also explicit training for rapid adaptation under language shift.

4. Embodied control and robotics

In robotics, PVMs are used in at least three distinct ways: as open-vocabulary grounding modules, as frozen perceptual front ends for policies, and as robust visual backbones under sim-to-real or OOD shift. MOO exemplifies the first pattern. It uses OWL-ViT as a frozen open-vocabulary detector that maps an image observation $o$ and language query $q$ into score maps and boxes, reduces the chosen detection to a single center-pixel mask, and feeds that mask with RGB and task-template language into an RT-1-based manipulation policy. The VLM is frozen throughout training. With 106 training objects and 13,239 additional pick episodes, MOO is evaluated in 1,472 real-world trials and reports substantial gains on novel objects, including around 50% improvement on unseen objects relative to strong baselines (Stone et al., 2023). The system’s key design choice is to separate semantic grounding from control by passing only a coarse object-centric mask to the policy.

A broader control study shows that off-the-shelf frozen visual representations pretrained entirely on out-of-domain data can be highly competitive in imitation learning. Across Habitat, DeepMind Control, Adroit, and Franka Kitchen, any pre-trained visual representation outperforms random features and end-to-end training from scratch in the low-data regime; MoCo matches ground-truth features in Habitat, and a full-hierarchy representation combining layers 3, 4, and 5 is competitive with or better than ground-truth state features across the tested domains (Parisi et al., 2022). However, another large benchmark across 14 PVMs and 21 motor-control tasks shows that representation rankings depend strongly on the downstream learner: BC and visual-reward-function methods produce more stable rankings, while RL is highly variable and therefore unreliable as a universal evaluation protocol (Hu et al., 2023).

The sim-to-real question is more nuanced. A large empirical study across three robots and five tasks finds that simulation trends are predictive of real-world trends when the setup is aligned to hardware constraints, reporting an SRCC of 0.895 and, after basic alignment, 0.929 (Silwal et al., 2023). The strongest result is indoor ImageNav: in a held-out real apartment, scratch achieves 10% success, whereas VC-1 Large achieves 90% zero-shot success (Silwal et al., 2023). This is evidence that PVM-backed perception can enable real deployment, but it does not imply that simulation-trained manipulation policies transfer equally well.

Model-based RL presents a further distinction between in-domain fit and robust generalization. A study integrating DINOv2 and CLIP into DreamerV3 finds little sample-efficiency benefit, largely confirming earlier negative conclusions, but reports that under severe visual shifts frozen or partially fine-tuned PVMs dramatically outperform a scratch CNN. In the hardest tabletop and driving shifts, full fine-tuning tends to collapse, whereas frozen or partially fine-tuned representations preserve much more of their pretrained invariance; partial fine-tuning often gives the best average performance under extreme shifts (Jones et al., 16 Sep 2025). This suggests that for vision-based control the relevant axis is not only adaptation strength but the plasticity-preservation balance of the visual backbone.

5. Robustness, safety, and model maintenance

PVMs create new attack surfaces because downstream models inherit not only useful features but also vulnerabilities encoded in the pretrained representation. In adversarial security, Downstream Transfer Attack (DTA) studies black-box attacks on fine-tuned models using only white-box access to the source pre-trained ViT. It defines Average Token Cosine Similarity (ATCS) at layer $k$,

$$\mathcal{L} = \alpha_1 \left(\mathcal{L}_+^{\text{mcls}} + \mathcal{L}_-^{\text{mcls}}\right) + \alpha_2 \mathcal{L}^{\text{cl}} + \alpha_3 \mathcal{L}^{\text{mim}} + \alpha_4 \mathcal{L}^{\text{mom}},$$0

and shows that lower ATCS strongly correlates with higher downstream attack success. Across three pretraining methods, three fine-tuning schemes, and ten downstream datasets, DTA reaches 93.11% average ASR on full fine-tuning, 95.22% on LoRA, and 95.32% on AdaptFormer (Zheng et al., 2024). The layer of maximal vulnerability depends on pretraining: shallow for MAE, middle for DINO, and deep for AugReg.

Supply-chain attacks operate one stage earlier. Pre-trained Trojan injects a backdoor during pretraining so that the infected backbone transfers the trigger across downstream tasks. Two ingredients are central: a stylized trigger carrying class-specific low-level texture cues and a context-free poisoning pipeline that pastes the trigger on blank backgrounds to create a direct shortcut from trigger to target class. With only 0.1% poisoning, the method reports 88.20% $$\mathcal{L} = \alpha_1 \left(\mathcal{L}_+^{\text{mcls}} + \mathcal{L}_-^{\text{mcls}}\right) + \alpha_2 \mathcal{L}^{\text{cl}} + \alpha_3 \mathcal{L}^{\text{mim}} + \alpha_4 \mathcal{L}^{\text{mom}},$$1 ASR on downstream ImageNet object detection; on COCO, targeted downstream detection and instance-segmentation ASRs reach about 99% for some targets (Liu et al., 2023). This shows that backdoors can be inherited at the representation level rather than only at the task-head level.

PVM maintenance also includes deliberate removal of knowledge. Continual Forgetting for Pre-trained Vision Models introduces GS-LoRA, which attaches task-specific LoRA modules to Transformer FFN layers and applies group sparse regularization so that only a subset of blocks is edited for each forgetting request. In face recognition, GS-LoRA tunes 1.28% of parameters; in object detection, 0.62%. It manages to forget specific classes with minimal impact on retained classes and substantially outperforms regularization and rehearsal baselines in both single-step and continual forgetting (Zhao et al., 2024). This suggests that selective erasure in large PVMs can be treated as a sparse adapter-editing problem rather than as full retraining.

Safety-critical deployment adds a different maintenance problem: intervention rather than deletion. A study of latent safety filters uses pretrained visual representations as backbones for failure classifiers, HJ reachability-based safety critics, and latent world models. It finds that fine-tuned PVRs usually reduce violations more effectively than frozen ones, that DINO-CLS is the most consistently strong HJ backbone, and that critic-only switching is often preferable unless the chosen latent space is especially suitable for world modeling. Reported inference times are below $$\mathcal{L} = \alpha_1 \left(\mathcal{L}_+^{\text{mcls}} + \mathcal{L}_-^{\text{mcls}}\right) + \alpha_2 \mathcal{L}^{\text{cl}} + \alpha_3 \mathcal{L}^{\text{mim}} + \alpha_4 \mathcal{L}^{\text{mom}},$$2 s, with model sizes up to 611 MB, making these filters practical at 50 Hz (Tabbara et al., 18 Sep 2025). The broader point is that PVMs can support not only nominal control but also safety infrastructure.

6. Open problems and research directions

Survey work identifies several unresolved technical issues. Stronger image-text alignment remains a central challenge; existing masking strategies often operate at the input or task level rather than aligning embedding structure directly. Pretraining strategy is still underexplored, especially multitask synergy, dataset selection, and objective composition. Evaluation is also immature: many systems are judged only through expensive downstream tasks, while intrinsic diagnostics for representation quality, transfer readiness, or alignment remain limited (Long et al., 2022). In modern multimodal systems, the visual encoder acts as a semantic backbone, but it also inherits failure modes such as hallucination, outdated knowledge, bias of concept association, and confusion on compositional concepts; proposed remedies include deeper cross-modal interaction, hard negatives, stronger visual encoders, and combining CLIP-style features with vision-only self-supervised features (Qi et al., 2024).

A second open problem is that there is no universally best PVM. The optimal choice depends on data distribution, task geometry, and downstream learner. For robotics, object-centric or part-centric structure appears decisive, but the appropriate granularity varies: scene-centric data favor navigation, while ego-centric data can favor manipulation, and part-level slots may outperform holistic object grouping for action-conditioned tasks (Wen et al., 10 Mar 2025). For control evaluation, BC and visual-reward-function methods probe different latent properties and often rank models differently (Hu et al., 2023). This suggests that future PVM design must be explicitly task-conditional rather than benchmark-monolithic.

A third direction concerns adaptation ability itself. Cross-lingual PVLM transfer improves when the model is trained not only to represent multimodal structure but also to adapt rapidly via meta-learning (Hu et al., 2023). Continual segmentation improves when classifier drift is controlled rather than when ever more elaborate anti-forgetting machinery is added (Zhang et al., 6 Aug 2025). Continual forgetting becomes feasible when erasure is treated as sparse low-rank editing rather than wholesale retraining (Zhao et al., 2024). These results collectively suggest that the next phase of PVM research will be less about a single dominant backbone and more about controllable, auditable, and task-aware adaptation of large pretrained representations.