---
title: Pre-trained Vision Models (PVRs)
url: https://www.emergentmind.com/topics/pre-trained-vision-models-pvrs
type: topic
---

# Pre-trained Vision Models (PVRs)

Searching arXiv for recent and foundational papers on pre-trained vision models and related vision-language adaptation.
arxiv_search(query="pre-trained vision models vision-language prompt tuning CLIP continual learning robotics survey", max_results=10)
Pre-trained vision models (PVRs) are visual backbones trained in advance on large-scale image, video, or image–text corpora and then reused as general-purpose feature extractors or adaptation substrates in downstream systems. Across the literature considered here, the term spans supervised and self-supervised vision backbones, vision encoders embedded in vision–language pre-trained models such as CLIP, and task-specialized variants trained for domains such as robotics or visual place recognition. A recurring theme is that the utility of a PVR depends not only on pre-training scale or architecture, but also on the adaptation interface, downstream objective, data distribution, and whether the representation preserves the structural properties required by the target problem, such as cross-modal alignment, object-centricity, trajectory-level similarity, or safety-relevant latent geometry [2306.11400; 2304.04591; 2303.18240; 2503.06960].

## 1. Definitions and conceptual scope

In the cited work, a pre-trained vision model is generally a visual encoder trained prior to the downstream task and then reused either as a frozen feature map \(z = f(x)\), as the vision branch of a vision–language model, or as a backbone augmented by prompts, adapters, routers, or task-specific heads [2304.04591; 2303.18240; 2412.08158]. In motor control, the canonical abstraction is a frozen encoder \(f\) mapping observations \(o_t\) to features \(z_t = f(o_t)\), followed by a lightweight policy head \(\pi\) producing actions \(a_t = \pi(z_t)\) [2304.04591]. In visual reinforcement learning, the same pattern is used with a frozen PVR \(\phi\) providing latent vectors \(\phi(x_t)\) to actor–critic networks [2407.17238]. In visual place recognition, the PVR is a Vision Transformer encoder whose class token is projected into a global descriptor and whose decoder can be repurposed into a pair classifier [2410.06614].

The concept extends naturally to vision–language systems. Large vision–language pre-trained models such as CLIP, ALIGN, and SimVLM replace the closed-set paradigm of purely visual supervised training with a shared image–text embedding space, in which the image encoder itself functions as a powerful pre-trained vision model [2306.11400; 2412.08158]. In CLIP, the text encoder \(\mathcal{T}\) and image encoder \(\mathcal{V}\) map both modalities into a common space of dimension \(d_c\), with textual representation
\[
z = w^{eos}_K \circ TextProj
\]
and image representation
\[
x = c_K \circ ImageProj,
\]
where the image encoder may be viewed as a ViT backbone trained under cross-modal supervision rather than conventional classification supervision [2306.11400].

Several papers also use broader terminology such as vision foundation models (VFMs) or pre-trained visual representations (PVRs), but the operational meaning remains similar: a reusable visual representation learned at scale, often frozen during downstream learning, sometimes with lightweight adaptation [2510.05213; 2303.18240; 2310.02219]. This suggests that “PVR” is best understood as a functional category rather than a single architectural family.

## 2. Architectural families and pre-training objectives

The surveyed literature spans CNNs, region-based encoders, Vision Transformers, and multimodal encoders. CNN backbones such as ResNet-50 appear in both classical and robotic settings, while ViTs dominate recent work in vision–language learning, masked image modeling, and task-specific pre-training [2412.08158; 2304.04591; 2503.06960]. Region-based encoders remain implicit in older vision–language pipelines such as BUTD and VinVL, where object detectors generate region features \(\mathbf{r}_1,\dots,\mathbf{r}_k\), whereas patch-based ViTs divide the image into tokens processed by self-attention
\[
\text{Attn}(Q,K,V)=\mathrm{softmax}\Big(\frac{QK^\top}{\sqrt{d}}\Big)V
\]
and produce a global visual embedding \(\mathbf{v}\) via a [CLS] token or pooling [2412.08158].

The principal pre-training objectives differ in the invariances and structures they induce. Contrastive image–text learning, exemplified by CLIP, optimizes a symmetric loss over image–text pairs,
\[
\mathcal{L}_{\text{contrast}} =
\frac{1}{N} \sum_{i=1}^N \Big[
- \log \frac{\exp( \mathrm{sim}(\mathbf{v}_i,\mathbf{t}_i)/\tau)}{\sum_{j=1}^N \exp(\mathrm{sim}(\mathbf{v}_i,\mathbf{t}_j)/\tau)}
- \log \frac{\exp( \mathrm{sim}(\mathbf{t}_i,\mathbf{v}_i)/\tau)}{\sum_{j=1}^N \exp(\mathrm{sim}(\mathbf{t}_i,\mathbf{v}_j)/\tau)}
\Big],
\]
creating a joint embedding space suitable for zero-shot classification, retrieval, and open-vocabulary detection [2412.08158]. Masked image modeling and masked autoencoding instead reconstruct masked pixels or patches, as in
\[
\mathcal{L}_{\text{MAE}} =
\frac{1}{|M|}\sum_{i\in M}\|x_i-\hat{x}_i\|_2^2,
\]
and are central to MVP, VC-1, MAE, and Pair-VPR [2303.18240; 2410.06614]. Self-distillation and clustering-based methods such as DINO and iBOT emphasize semantically structured embeddings and, in some settings, stronger object-centricity than MAE [2503.06960].

Multi-objective vision–language pre-training combines several of these ingredients. BLIP’s MED architecture uses image–text contrastive learning, image–text matching,
\[
\mathcal{L}_{\text{ITM}} = - \sum_i y_i \log p_i + (1-y_i)\log(1-p_i),
\]
and image-conditioned language modeling
\[
\mathcal{L}_{\text{LM}} = - \sum_t \log p_\theta(w_t \mid w_{<t}, \mathbf{v}),
\]
to support both recognition and generation [2412.08158]. This suggests that PVR design has shifted from single-objective feature extraction toward representations explicitly structured for alignment, generation, and transfer.

## 3. Adaptation interfaces: prompts, adapters, routing, and frozen backbones

A major line of work concerns how PVRs should be adapted without destructive end-to-end fine-tuning. In CLIP-like systems, prompt tuning emerged as a parameter-efficient adaptation mechanism, but uni-modal prompting can be sub-optimal because it moves one modality while freezing the other, weakening the pre-trained cross-modal alignment [2306.11400]. MuDPT addresses this by introducing deep textual prompts
\[
T \in \mathbb{R}^{L \times n \times d_t}
\]
and deep visual prompts
\[
V \in \mathbb{R}^{L \times n \times d_v},
\]
together with an Injection Model that produces cross-modality prompts \(T',V'\) and fuses them as
\[
\hat{T}, \hat{V} = T + V',\; V + T'.
\]
Only the prompts and Injection Model are trained; CLIP’s encoders and projection layers remain frozen [2306.11400]. On 16-shot recognition across 11 datasets, MuDPT reaches \(81.38\%\) average accuracy versus \(73.18\%\) for CoOp and \(75.07\%\) for CoCoOp, while preserving base-to-new generalization better than text-only prompting [2306.11400].

Class-incremental learning yields a different adaptation picture. A study of CLIP for continual learning compares a Linear Adapter, Self-attention Adapter, and Prompt Tuning, all with a frozen CLIP backbone [2310.20348]. The Linear Adapter \(A_i = W I_i\), trained with logits \(A_i^\top B_t\), consistently performs best, especially when combined with a drift-based parameter retention rule
\[
W_2^{\prime,i} =
\begin{cases}
W_1^i, & \text{if } \Delta W^i < \beta(\gamma) \\
W_2^i, & \text{if } \Delta W^i \ge \beta(\gamma),
\end{cases}
\]
where \(\Delta W^i = |W_1^i - W_2^i|\) [2310.20348]. This paper explicitly finds that the simplest solution, a single Linear Adapter with parameter retention, gives the best results.

Robotics work pushes parameter-efficient adaptation further toward structured conditional computation. VER distills multiple VFMs into a Vision Expert Library and trains only a lightweight router, fewer than \(0.4\%\) of parameters, for downstream robot tasks [2510.05213]. At the MoE layer level,
\[
y = \sum_{l=1}^{L} \mathcal{R}_i^n(x,l)\cdot \mathcal{E}_l^n(x),
\]
with Top-\(K\) routing and later patchwise expert routing plus Curriculum Top-\(K\) Annealing
\[
K(s)=\max\Big(K_{\min},\,\Big\lfloor L-(L-K_{\min})\cdot \frac{s}{S}\Big\rfloor\Big).
\]
VER’s results across 17 robotic tasks indicate that routing among multiple distilled experts is more effective than a single static representation [2510.05213]. A plausible implication is that future PVR adaptation may increasingly resemble conditional expert selection rather than uniform fine-tuning.

Continual modification of PVRs also appears in forgetting rather than learning. GS-LoRA addresses continual forgetting by attaching LoRA modules to FFN layers and applying group sparsity
\[
\mathcal{L}_{structure}=\sum_{\ell=1}^G \left(\|\mathbf{B}^{(\ell)}_t\|_F + \|\mathbf{A}^{(\ell)}_t\|_F\right),
\]
so that unwanted knowledge can be removed while retaining the rest [2403.11530]. This broadens the adaptation landscape from specialization to governance and selective erasure.

## 4. Data regime, object-centricity, and domain-specific pre-training

A recurrent conclusion is that the effectiveness of a PVR depends strongly on the interaction between pre-training objective and data distribution. In embodied AI, CortexBench evaluates CLIP, R3M, MVP, VIP, and MAE-trained ViTs across 17 tasks spanning locomotion, dexterous manipulation, navigation, and mobile manipulation [2303.18240]. No PVR is universally dominant: CLIP is best on ObjectNav, R3M on Adroit, MetaWorld, and DMC, and MVP on TriFinger, ImageNav, and MobilePick, while VC-1 achieves the best average Mean Success and Mean Rank without dominating every benchmark [2303.18240]. Scaling data and model size helps on average but not uniformly; adding new data domains such as navigation or ImageNet improves some tasks while leaving task-specific optima unchanged [2303.18240].

A more explicitly data-centric account is developed by SlotMIM. This work argues that the key to successful PVMs for robot learning is the ability to form object-centric representations from non-object-centric (NOC) data such as COCO+, web data, or ego-centric video [2503.06960]. DINO and iBOT outperform MAE on many tasks when trained on object-centric ImageNet, but degrade sharply on NOC data, and this degradation correlates with reduced object-centricity as measured by VOC Jaccard and manipulation success [2503.06960]. SlotMIM introduces a semantic bottleneck by reducing prototype count, adds cross-view patch consistency,
\[
\mathcal{L}_\text{patch}^\text{cross}(x^1,x^2)
= -\frac{1}{|\mathcal{P}|}\sum_{(i,j)\in\mathcal{P}}\sum_{c=1}^C q_{\xi,i,c}^2 \log \tilde{p}_{\theta,j,c}^1,
\]
and defines slot-level contrastive learning over pooled object slots [2503.06960]. Across recognition, scene understanding, navigation, and robot manipulation, SlotMIM improves transfer and shows superior data efficiency when scaled [2503.06960]. This suggests that object-centricity is not merely a desirable interpretive property but a concrete transfer predictor.

Task-specific pre-training can be even more specialized. Pair-VPR departs from generic ImageNet initialization by using place-aware Siamese masked image modeling on VPR datasets, sampling positive pairs from the same place and reconstructing one masked view using the other [2410.06614]. The Stage 1 loss
\[
\mathcal{L}(I_a,I_b)
=
\frac{1}{|p_a/\tilde{p}_a|}
\sum_{p_a^i \in p_a/\tilde{p}_a}
\|\hat{p}_a^i-p_a^i\|^2
\]
produces an encoder and decoder subsequently reused for retrieval and pair classification [2410.06614]. Pair-VPR-s with ViT-B reaches \(93.7\) R@1 on MSLS-Val, \(98.1\) on Tokyo24/7, and \(84.2\) on Nordland, while larger encoders push Tokyo24/7 to \(100\%\) R@1 [2410.06614]. This indicates that domain-aware pre-training can surpass strong generic backbones when the task requires specific invariances, here place invariance under viewpoint and season change.

## 5. Downstream learning regimes: control, recognition, safety, and open-world grounding

The value of a PVR is filtered by the downstream learning algorithm. A systematic study of 14 PVRs in motor control compares reinforcement learning with DrQ-v2, behavior cloning, and visual reward function learning via regularized optimal transport [2304.04591]. The central finding is that the effectiveness of pre-training is highly dependent on the downstream policy learning algorithm [2304.04591]. RL-based rankings are unstable across environments and seeds, with broad confidence intervals and cases where CLIP is strong on Meta-World but worst on Robosuite [2304.04591]. By contrast, BC is stable and correlates strongly with linear probing of ground-truth environment states; on Franka-Kitchen, linear probe loss correlates with BC IQM success rate with Pearson \(r=-0.78\) and Spearman \(\rho=-0.73\) [2304.04591]. VRF evaluation produces yet another ranking, favoring joint-embedding models such as MoCo v2, VICRegL, DINO, and CLIP, and exposing MAE’s near-zero utility for similarity-based reward shaping [2304.04591]. A plausible implication is that “representation quality” is not a scalar property but a relation between feature geometry and the downstream optimization problem.

Reinforcement-learning-specific studies refine this picture. In visual RL, frozen PVRs reduce replay buffer size by more than \(90\%\) and lower the number of trainable parameters, but task performance remains task-dependent [2407.17238]. On Metaworld Push-v2, DRM trained from scratch and a truncated ResNet18 PVR outperform DINOv2 and Visual Cortex, whereas on Drawer-Open-v2 DINOv2 surpasses both [2407.17238]. Dormant ratio is strongly correlated with success, and failed seeds often coincide with high dormant ratio [2407.17238]. In model-based RL with DreamerV3, pre-trained vision models do not improve sample efficiency, but under severe distribution shifts they outperform a baseline trained from scratch, and partial fine-tuning of the last quarter of layers gives the best average robustness [2509.12531]. In table-top manipulation and autonomous driving, frozen or partially fine-tuned PVMs retain stable latent structure under hard texture, weather, and lighting shifts, whereas full fine-tuning and scratch training collapse [2509.12531]. This directly contradicts any simple claim that pre-training is either categorically effective or categorically ineffective in MBRL.

Safety-critical control introduces yet another downstream requirement. Latent safety filters use PVRs as backbones for failure classifiers, HJ-reachability critics, and latent world models [2509.14758]. Observations \(o_t\) are mapped to latents \(z_t=\text{enc}_\theta(o_t)\), failure sets are defined through \(h(z)\), and the HJ critic is trained with
\[
L(\theta)=\mathbb{E}_{(s_t,a_t,s_{t+1})\sim D}
\big[(Q_\theta(s_t,a_t)-y_t)^2\big]
\]
where
\[
y_t = (1-\gamma)h(s_t)+\gamma \min\{h(s_t),\max_a Q_\theta(s_{t+1},a)\}.
\]
Fine-tuning the PVR generally reduces safety violations, but no single PVR dominates; DINO-CLS is consistently strong, while the value of world-model lookahead versus critic-only switching depends on whether the PVR yields dynamics-friendly latents [2509.14758]. This suggests that downstream evaluation of PVRs for safety must include dynamic predictability, not only static classification quality.

Open-world robotics uses PVRs in a different way: as semantic interfaces between language and action. MOO uses frozen OWL-ViT as an open-vocabulary object detector and conditions an RT-1-style policy on the current image, a verb embedding, and an object-identifying mask derived from detected bounding box centers [2303.00905]. A single-channel mask encodes one object at value \(1.0\), or two objects at values \(1.0\) and \(0.5\), and is concatenated to the visual input [2303.00905]. The result is zero-shot manipulation of novel categories, including objects never seen in robot data, because semantic grounding is outsourced to the frozen VLM while control generalization is learned from demonstrations [2303.00905].

## 6. Risks, failure modes, and emerging governance questions

The literature repeatedly stresses that PVRs are not universally robust. In vision–language systems, surveys highlight hallucination, outdated knowledge, concept association bias, and compositional confusion [2412.08158; 2207.01772]. CLIP-like models may treat text as a bag of concepts, ignore syntax, and exhibit failures on composition and counting, which can propagate into downstream captioning, VQA, or open-vocabulary detection [2412.08158]. Generative pipelines can compound these errors when early-stage hallucinations contaminate subsequent reasoning modules [2412.08158].

Security work shows that vulnerabilities can transfer through the entire pretrain–finetune stack. Downstream Transfer Attack demonstrates that if an attacker has white-box access only to a pre-trained ViT, adversarial examples crafted by minimizing Average Token Cosine Similarity at selected vulnerable layers can fool downstream fine-tuned models with average attack success rate exceeding \(90\%\) across 10 datasets, 3 pre-training methods, and 3 fine-tuning schemes [2408.01705]. For classification, the attack optimizes perturbations within an \(\ell_\infty\) ball using
\[
x'_{t+1} =
\operatorname{Clip}_{x,\epsilon}\{x'_t+\eta \cdot \operatorname{sign}(\nabla_{x'_t}L_{\text{ATCS}})\},
\]
and similar transfer effects appear for detection and segmentation [2408.01705]. This reveals that downstream models inherit not only useful representations but also attack surfaces from their pre-trained backbones.

Selective erasure becomes a governance issue when pre-trained models encode private, biased, or undesirable knowledge. Continual forgetting formulates sequential deletion requests over PVRs and implements them with GS-LoRA attached to Transformer FFN layers [2403.11530]. The forgetting loss
\[
\mathcal{L}_{forget}
=
\text{ReLU}\Big(\text{BND} - \mathcal{L}\big(f_{M_{t-1}}(\mathcal{X}_{f_t}),\mathcal{Y}_{f_t}\big)\Big)
\]
increases loss on data to be forgotten while retention loss preserves performance on replayed retained classes [2403.11530]. The paper shows that specific classes can be forgotten in face recognition, object detection, and image classification with minimal collateral damage [2403.11530]. This indicates that the maintenance problem for PVRs now includes editing, deletion, and compliance, not only pre-training and transfer.

## 7. Synthesis and design principles

Across the cited work, several design principles recur. First, no universal PVR exists. Different pre-training objectives and data distributions induce different latent geometries, and these interact sharply with downstream algorithms, whether BC, VRF, RL, MBRL, HJ filtering, or open-vocabulary grounding [2304.04591; 2303.18240; 2509.14758]. Second, frozen backbones plus lightweight adaptation remain a strong default, but the correct adaptation unit varies by setting: prompts for multimodal alignment, linear adapters for continual learning, expert routers for robotics, LoRA groups for forgetting, or partial fine-tuning for MBRL [2306.11400; 2310.20348; 2510.05213; 2403.11530; 2509.12531]. Third, object-centricity, cross-modal alignment, and dynamics-friendly latent structure are not interchangeable; each predicts performance in some downstream families and failure in others [2503.06960; 2306.11400; 2509.14758].

A plausible implication is that the mature view of PVRs is no longer “pre-train once, reuse everywhere” in a naive sense. Rather, pre-training supplies a structured prior whose value depends on whether downstream adaptation preserves the right invariances and whether the representation geometry matches the operational demands of the task. For vision–language learning this may mean preserving shared embedding alignment; for robotics, retaining object-centric and temporally meaningful structure; for place recognition, learning place-aware invariances; and for safety, producing low-dimensional latents in which failure sets and dynamics remain tractable [2306.11400; 2503.06960; 2410.06614; 2509.14758].

In that sense, pre-trained vision models are best understood not as a monolithic class of universally superior backbones, but as reusable visual priors whose success depends on a joint design problem spanning pre-training objective, dataset composition, adaptation mechanism, downstream learning algorithm, and robustness constraints. The literature reviewed here consistently supports that broader interpretation [2303.18240; 2304.04591; 2412.08158].

Source: https://www.emergentmind.com/topics/pre-trained-vision-models-pvrs