Pre-trained Vision Models (PVRs)

• Pre-trained Vision Models (PVRs) are visual encoders trained on vast image or image–text corpora to serve as reusable backbones for diverse downstream applications.
• They utilize adaptation interfaces like prompts, adapters, and expert routers to efficiently tune models for tasks such as vision–language alignment and robotics.
• Their effectiveness depends on the interplay between pre-training objectives, data distributions, and specific downstream learning algorithms, ensuring tailored transfer capabilities.

Searching arXiv for recent and foundational papers on pre-trained vision models and related vision-language adaptation. arxiv_search(query="pre-trained vision models vision-language prompt tuning CLIP continual learning robotics survey", max_results=10) Pre-trained vision models (PVRs) are visual backbones trained in advance on large-scale image, video, or image–text corpora and then reused as general-purpose feature extractors or adaptation substrates in downstream systems. Across the literature considered here, the term spans supervised and self-supervised vision backbones, vision encoders embedded in vision–language pre-trained models such as CLIP, and task-specialized variants trained for domains such as robotics or visual place recognition. A recurring theme is that the utility of a PVR depends not only on pre-training scale or architecture, but also on the adaptation interface, downstream objective, data distribution, and whether the representation preserves the structural properties required by the target problem, such as cross-modal alignment, object-centricity, trajectory-level similarity, or safety-relevant latent geometry (Miao et al., 2023, Hu et al., 2023, Majumdar et al., 2023, Wen et al., 10 Mar 2025).

1. Definitions and conceptual scope

In the cited work, a pre-trained vision model is generally a visual encoder trained prior to the downstream task and then reused either as a frozen feature map $z = f(x)$, as the vision branch of a vision–LLM, or as a backbone augmented by prompts, adapters, routers, or task-specific heads (Hu et al., 2023, Majumdar et al., 2023, Qi et al., 2024). In motor control, the canonical abstraction is a frozen encoder $f$ mapping observations $o_t$ to features $z_t = f(o_t)$, followed by a lightweight policy head $\pi$ producing actions $a_t = \pi(z_t)$ (Hu et al., 2023). In visual reinforcement learning, the same pattern is used with a frozen PVR $\phi$ providing latent vectors $\phi(x_t)$ to actor–critic networks (Williams et al., 2024). In visual place recognition, the PVR is a Vision Transformer encoder whose class token is projected into a global descriptor and whose decoder can be repurposed into a pair classifier (Hausler et al., 2024).

The concept extends naturally to vision–language systems. Large vision–language pre-trained models such as CLIP, ALIGN, and SimVLM replace the closed-set paradigm of purely visual supervised training with a shared image–text embedding space, in which the image encoder itself functions as a powerful pre-trained vision model (Miao et al., 2023, Qi et al., 2024). In CLIP, the text encoder $\mathcal{T}$ and image encoder $\mathcal{V}$ map both modalities into a common space of dimension $f$0, with textual representation

$f$1

and image representation

$f$2

where the image encoder may be viewed as a ViT backbone trained under cross-modal supervision rather than conventional classification supervision (Miao et al., 2023).

Several papers also use broader terminology such as vision foundation models (VFMs) or pre-trained visual representations (PVRs), but the operational meaning remains similar: a reusable visual representation learned at scale, often frozen during downstream learning, sometimes with lightweight adaptation (Wang et al., 6 Oct 2025, Majumdar et al., 2023, Silwal et al., 2023). This suggests that “PVR” is best understood as a functional category rather than a single architectural family.

2. Architectural families and pre-training objectives

The surveyed literature spans CNNs, region-based encoders, Vision Transformers, and multimodal encoders. CNN backbones such as ResNet-50 appear in both classical and robotic settings, while ViTs dominate recent work in vision–language learning, masked image modeling, and task-specific pre-training (Qi et al., 2024, Hu et al., 2023, Wen et al., 10 Mar 2025). Region-based encoders remain implicit in older vision–language pipelines such as BUTD and VinVL, where object detectors generate region features $f$3, whereas patch-based ViTs divide the image into tokens processed by self-attention

$f$4

and produce a global visual embedding $f$5 via a [CLS] token or pooling (Qi et al., 2024).

The principal pre-training objectives differ in the invariances and structures they induce. Contrastive image–text learning, exemplified by CLIP, optimizes a symmetric loss over image–text pairs,

$f$6

creating a joint embedding space suitable for zero-shot classification, retrieval, and open-vocabulary detection (Qi et al., 2024). Masked image modeling and masked autoencoding instead reconstruct masked pixels or patches, as in

$f$7

and are central to MVP, VC-1, MAE, and Pair-VPR (Majumdar et al., 2023, Hausler et al., 2024). Self-distillation and clustering-based methods such as DINO and iBOT emphasize semantically structured embeddings and, in some settings, stronger object-centricity than MAE (Wen et al., 10 Mar 2025).

Multi-objective vision–language pre-training combines several of these ingredients. BLIP’s MED architecture uses image–text contrastive learning, image–text matching,

$f$8

and image-conditioned language modeling

$f$9

to support both recognition and generation (Qi et al., 2024). This suggests that PVR design has shifted from single-objective feature extraction toward representations explicitly structured for alignment, generation, and transfer.

3. Adaptation interfaces: prompts, adapters, routing, and frozen backbones

A major line of work concerns how PVRs should be adapted without destructive end-to-end fine-tuning. In CLIP-like systems, prompt tuning emerged as a parameter-efficient adaptation mechanism, but uni-modal prompting can be sub-optimal because it moves one modality while freezing the other, weakening the pre-trained cross-modal alignment (Miao et al., 2023). MuDPT addresses this by introducing deep textual prompts

$o_t$0

and deep visual prompts

$o_t$1

together with an Injection Model that produces cross-modality prompts $o_t$2 and fuses them as

$o_t$3

Only the prompts and Injection Model are trained; CLIP’s encoders and projection layers remain frozen (Miao et al., 2023). On 16-shot recognition across 11 datasets, MuDPT reaches $o_t$4 average accuracy versus $o_t$5 for CoOp and $o_t$6 for CoCoOp, while preserving base-to-new generalization better than text-only prompting (Miao et al., 2023).

Class-incremental learning yields a different adaptation picture. A study of CLIP for continual learning compares a Linear Adapter, Self-attention Adapter, and Prompt Tuning, all with a frozen CLIP backbone (Liu et al., 2023). The Linear Adapter $o_t$7, trained with logits $o_t$8, consistently performs best, especially when combined with a drift-based parameter retention rule

$o_t$9

where $z_t = f(o_t)$0 (Liu et al., 2023). This paper explicitly finds that the simplest solution, a single Linear Adapter with parameter retention, gives the best results.

Robotics work pushes parameter-efficient adaptation further toward structured conditional computation. VER distills multiple VFMs into a Vision Expert Library and trains only a lightweight router, fewer than $z_t = f(o_t)$1 of parameters, for downstream robot tasks (Wang et al., 6 Oct 2025). At the MoE layer level,

$z_t = f(o_t)$2

with Top-$z_t = f(o_t)$3 routing and later patchwise expert routing plus Curriculum Top-$z_t = f(o_t)$4 Annealing

$z_t = f(o_t)$5

VER’s results across 17 robotic tasks indicate that routing among multiple distilled experts is more effective than a single static representation (Wang et al., 6 Oct 2025). A plausible implication is that future PVR adaptation may increasingly resemble conditional expert selection rather than uniform fine-tuning.

Continual modification of PVRs also appears in forgetting rather than learning. GS-LoRA addresses continual forgetting by attaching LoRA modules to FFN layers and applying group sparsity

$z_t = f(o_t)$6

so that unwanted knowledge can be removed while retaining the rest (Zhao et al., 2024). This broadens the adaptation landscape from specialization to governance and selective erasure.

4. Data regime, object-centricity, and domain-specific pre-training

A recurrent conclusion is that the effectiveness of a PVR depends strongly on the interaction between pre-training objective and data distribution. In embodied AI, CortexBench evaluates CLIP, R3M, MVP, VIP, and MAE-trained ViTs across 17 tasks spanning locomotion, dexterous manipulation, navigation, and mobile manipulation (Majumdar et al., 2023). No PVR is universally dominant: CLIP is best on ObjectNav, R3M on Adroit, MetaWorld, and DMC, and MVP on TriFinger, ImageNav, and MobilePick, while VC-1 achieves the best average Mean Success and Mean Rank without dominating every benchmark (Majumdar et al., 2023). Scaling data and model size helps on average but not uniformly; adding new data domains such as navigation or ImageNet improves some tasks while leaving task-specific optima unchanged (Majumdar et al., 2023).

A more explicitly data-centric account is developed by SlotMIM. This work argues that the key to successful PVMs for robot learning is the ability to form object-centric representations from non-object-centric (NOC) data such as COCO+, web data, or ego-centric video (Wen et al., 10 Mar 2025). DINO and iBOT outperform MAE on many tasks when trained on object-centric ImageNet, but degrade sharply on NOC data, and this degradation correlates with reduced object-centricity as measured by VOC Jaccard and manipulation success (Wen et al., 10 Mar 2025). SlotMIM introduces a semantic bottleneck by reducing prototype count, adds cross-view patch consistency,

$z_t = f(o_t)$7

and defines slot-level contrastive learning over pooled object slots (Wen et al., 10 Mar 2025). Across recognition, scene understanding, navigation, and robot manipulation, SlotMIM improves transfer and shows superior data efficiency when scaled (Wen et al., 10 Mar 2025). This suggests that object-centricity is not merely a desirable interpretive property but a concrete transfer predictor.

Task-specific pre-training can be even more specialized. Pair-VPR departs from generic ImageNet initialization by using place-aware Siamese masked image modeling on VPR datasets, sampling positive pairs from the same place and reconstructing one masked view using the other (Hausler et al., 2024). The Stage 1 loss

$z_t = f(o_t)$8

produces an encoder and decoder subsequently reused for retrieval and pair classification (Hausler et al., 2024). Pair-VPR-s with ViT-B reaches $z_t = f(o_t)$9 R@1 on MSLS-Val, $\pi$0 on Tokyo24/7, and $\pi$1 on Nordland, while larger encoders push Tokyo24/7 to $\pi$2 R@1 (Hausler et al., 2024). This indicates that domain-aware pre-training can surpass strong generic backbones when the task requires specific invariances, here place invariance under viewpoint and season change.

5. Downstream learning regimes: control, recognition, safety, and open-world grounding

The value of a PVR is filtered by the downstream learning algorithm. A systematic study of 14 PVRs in motor control compares reinforcement learning with DrQ-v2, behavior cloning, and visual reward function learning via regularized optimal transport (Hu et al., 2023). The central finding is that the effectiveness of pre-training is highly dependent on the downstream policy learning algorithm (Hu et al., 2023). RL-based rankings are unstable across environments and seeds, with broad confidence intervals and cases where CLIP is strong on Meta-World but worst on Robosuite (Hu et al., 2023). By contrast, BC is stable and correlates strongly with linear probing of ground-truth environment states; on Franka-Kitchen, linear probe loss correlates with BC IQM success rate with Pearson $\pi$3 and Spearman $\pi$4 (Hu et al., 2023). VRF evaluation produces yet another ranking, favoring joint-embedding models such as MoCo v2, VICRegL, DINO, and CLIP, and exposing MAE’s near-zero utility for similarity-based reward shaping (Hu et al., 2023). A plausible implication is that “representation quality” is not a scalar property but a relation between feature geometry and the downstream optimization problem.

Reinforcement-learning-specific studies refine this picture. In visual RL, frozen PVRs reduce replay buffer size by more than $\pi$5 and lower the number of trainable parameters, but task performance remains task-dependent (Williams et al., 2024). On Metaworld Push-v2, DRM trained from scratch and a truncated ResNet18 PVR outperform DINOv2 and Visual Cortex, whereas on Drawer-Open-v2 DINOv2 surpasses both (Williams et al., 2024). Dormant ratio is strongly correlated with success, and failed seeds often coincide with high dormant ratio (Williams et al., 2024). In model-based RL with DreamerV3, pre-trained vision models do not improve sample efficiency, but under severe distribution shifts they outperform a baseline trained from scratch, and partial fine-tuning of the last quarter of layers gives the best average robustness (Jones et al., 16 Sep 2025). In table-top manipulation and autonomous driving, frozen or partially fine-tuned PVMs retain stable latent structure under hard texture, weather, and lighting shifts, whereas full fine-tuning and scratch training collapse (Jones et al., 16 Sep 2025). This directly contradicts any simple claim that pre-training is either categorically effective or categorically ineffective in MBRL.

Safety-critical control introduces yet another downstream requirement. Latent safety filters use PVRs as backbones for failure classifiers, HJ-reachability critics, and latent world models (Tabbara et al., 18 Sep 2025). Observations $\pi$6 are mapped to latents $\pi$7, failure sets are defined through $\pi$8, and the HJ critic is trained with

$\pi$9

where

$a_t = \pi(z_t)$0

Fine-tuning the PVR generally reduces safety violations, but no single PVR dominates; DINO-CLS is consistently strong, while the value of world-model lookahead versus critic-only switching depends on whether the PVR yields dynamics-friendly latents (Tabbara et al., 18 Sep 2025). This suggests that downstream evaluation of PVRs for safety must include dynamic predictability, not only static classification quality.

Open-world robotics uses PVRs in a different way: as semantic interfaces between language and action. MOO uses frozen OWL-ViT as an open-vocabulary object detector and conditions an RT-1-style policy on the current image, a verb embedding, and an object-identifying mask derived from detected bounding box centers (Stone et al., 2023). A single-channel mask encodes one object at value $a_t = \pi(z_t)$1, or two objects at values $a_t = \pi(z_t)$2 and $a_t = \pi(z_t)$3, and is concatenated to the visual input (Stone et al., 2023). The result is zero-shot manipulation of novel categories, including objects never seen in robot data, because semantic grounding is outsourced to the frozen VLM while control generalization is learned from demonstrations (Stone et al., 2023).

6. Risks, failure modes, and emerging governance questions

The literature repeatedly stresses that PVRs are not universally robust. In vision–language systems, surveys highlight hallucination, outdated knowledge, concept association bias, and compositional confusion (Qi et al., 2024, Nguyen et al., 2022). CLIP-like models may treat text as a bag of concepts, ignore syntax, and exhibit failures on composition and counting, which can propagate into downstream captioning, VQA, or open-vocabulary detection (Qi et al., 2024). Generative pipelines can compound these errors when early-stage hallucinations contaminate subsequent reasoning modules (Qi et al., 2024).

Security work shows that vulnerabilities can transfer through the entire pretrain–finetune stack. Downstream Transfer Attack demonstrates that if an attacker has white-box access only to a pre-trained ViT, adversarial examples crafted by minimizing Average Token Cosine Similarity at selected vulnerable layers can fool downstream fine-tuned models with average attack success rate exceeding $a_t = \pi(z_t)$4 across 10 datasets, 3 pre-training methods, and 3 fine-tuning schemes (Zheng et al., 2024). For classification, the attack optimizes perturbations within an $a_t = \pi(z_t)$5 ball using

$a_t = \pi(z_t)$6

and similar transfer effects appear for detection and segmentation (Zheng et al., 2024). This reveals that downstream models inherit not only useful representations but also attack surfaces from their pre-trained backbones.

Selective erasure becomes a governance issue when pre-trained models encode private, biased, or undesirable knowledge. Continual forgetting formulates sequential deletion requests over PVRs and implements them with GS-LoRA attached to Transformer FFN layers (Zhao et al., 2024). The forgetting loss

$a_t = \pi(z_t)$7

increases loss on data to be forgotten while retention loss preserves performance on replayed retained classes (Zhao et al., 2024). The paper shows that specific classes can be forgotten in face recognition, object detection, and image classification with minimal collateral damage (Zhao et al., 2024). This indicates that the maintenance problem for PVRs now includes editing, deletion, and compliance, not only pre-training and transfer.

7. Synthesis and design principles

Across the cited work, several design principles recur. First, no universal PVR exists. Different pre-training objectives and data distributions induce different latent geometries, and these interact sharply with downstream algorithms, whether BC, VRF, RL, MBRL, HJ filtering, or open-vocabulary grounding (Hu et al., 2023, Majumdar et al., 2023, Tabbara et al., 18 Sep 2025). Second, frozen backbones plus lightweight adaptation remain a strong default, but the correct adaptation unit varies by setting: prompts for multimodal alignment, linear adapters for continual learning, expert routers for robotics, LoRA groups for forgetting, or partial fine-tuning for MBRL (Miao et al., 2023, Liu et al., 2023, Wang et al., 6 Oct 2025, Zhao et al., 2024, Jones et al., 16 Sep 2025). Third, object-centricity, cross-modal alignment, and dynamics-friendly latent structure are not interchangeable; each predicts performance in some downstream families and failure in others (Wen et al., 10 Mar 2025, Miao et al., 2023, Tabbara et al., 18 Sep 2025).

A plausible implication is that the mature view of PVRs is no longer “pre-train once, reuse everywhere” in a naive sense. Rather, pre-training supplies a structured prior whose value depends on whether downstream adaptation preserves the right invariances and whether the representation geometry matches the operational demands of the task. For vision–language learning this may mean preserving shared embedding alignment; for robotics, retaining object-centric and temporally meaningful structure; for place recognition, learning place-aware invariances; and for safety, producing low-dimensional latents in which failure sets and dynamics remain tractable (Miao et al., 2023, Wen et al., 10 Mar 2025, Hausler et al., 2024, Tabbara et al., 18 Sep 2025).

In that sense, pre-trained vision models are best understood not as a monolithic class of universally superior backbones, but as reusable visual priors whose success depends on a joint design problem spanning pre-training objective, dataset composition, adaptation mechanism, downstream learning algorithm, and robustness constraints. The literature reviewed here consistently supports that broader interpretation (Majumdar et al., 2023, Hu et al., 2023, Qi et al., 2024).