Visual Pretraining: Methods & Applications

• Visual Pretraining is a design space that encompasses various proxy tasks (e.g., image-text, video, document) to learn transferable semantic and structural visual representations.
• It employs methodologies such as contrastive learning, masked modeling, and autoregressive token prediction to capture spatial, temporal, and cross-modal invariances.
• Its applications span vision-language models, video analysis, and document understanding, enhancing downstream transfer effectiveness and reasoning accuracy.

Visual pretraining denotes a family of learning procedures in which a model is optimized on visual or visually grounded proxy tasks before downstream adaptation. In the surveyed literature, the term spans image-only masked modeling, video self-supervision, vision-language pre-training, autoregressive modeling of spatial annotations, instruction-driven end-to-end adaptation of visual backbones, and, in a more recent and narrower sense, direct pretraining on rendered document pages for language intelligence rather than on extracted text alone (Chen et al., 2022, Zhang et al., 10 Jul 2026). Across these formulations, the common premise is that pretraining should induce reusable structure—semantic, spatial, temporal, or cross-modal—that is not tied to a single downstream label space.

1. Scope and major formulations

The literature treats visual pretraining as an umbrella rather than a single recipe. In vision-language work, pre-training is performed on paired image-text or video-text data so that models learn unified representations of both modalities (Chen et al., 2022). In image-only work, masked image modeling is a prominent formulation, with variants differing mainly in the target space to be reconstructed or aligned. In video, self-supervised objectives are used to expose models to motion, object permanence, and temporal dynamics. Other work makes the target explicitly structural: VisorGPT learns a “visual prior” by tokenizing bounding boxes, human pose, and instance masks into sequences and fitting an autoregressive decoder to their likelihood (Xie et al., 2023). Recent document-centric work uses the label “Visual Pretraining” for a continued pretraining stage in which a LLM predicts visual latents from rendered pages instead of learning solely from text extracted from those pages (Zhang et al., 10 Jul 2026).

Formulation	Pretraining signal	Representative work
Vision-language pre-training	ITC, MLM, MVM, ITM on paired data	(Chen et al., 2022, Li et al., 2021)
Multimodality-guided image pretraining	Cosine alignment to frozen CLIP vision features on masked patches	(Wei et al., 2022)
Video self-supervision	Masked reconstruction or contrastive learning across frames	(Sun et al., 2023, Parthasarathy et al., 2022)
Explicit spatial-prior learning	Autoregressive likelihood over discretized annotations	(Xie et al., 2023)
Document-image pretraining for LLMs	Mixed text CE and next-visual-latent InfoNCE on raw pages	(Zhang et al., 10 Jul 2026)
Instruction-driven domain adaptation	End-to-end SFT with dropped visual tokens via VRL	(Li et al., 22 Sep 2025)

This diversity matters because “visual” can refer to several distinct sources of supervision: pixel patches, region features, slot tokens, sequence-encoded layouts, page renderings, or visually conditioned language responses. Taken together, these works suggest that VP is best understood as a design space of pretraining signals and tokenizations rather than as a fixed architectural category.

2. Objectives and representational targets

A central distinction among VP methods is the choice of proxy objective. The vision-language survey identifies four widely used objectives: image-text contrastive learning, masked language modeling, masked vision modeling, and image-text matching (Chen et al., 2022). These objectives induce different invariances. ITC emphasizes global cross-modal alignment; MLM and MVM encourage conditional reconstruction; ITM enforces pair-level compatibility.

Several later systems specialize these targets. VisorGPT formalizes spatial prior learning as left-to-right likelihood maximization over a sequence of discretized annotation tokens:

$$\mathcal{L}(\Theta)=\sum_{i=1}^{n}\log p_\Theta(u_i \mid u_{<i}) .$$

The model uses no auxiliary losses; the objective is purely autoregressive (Xie et al., 2023). By contrast, the document-centric VP formulation combines standard text next-token cross-entropy with a next-visual-latent contrastive loss:

$$L=\lambda_{\text{text}}L_{CE}+\lambda_{\text{vis}}L_{VP},$$

with $\lambda_{\text{vis}}$ typically set to $0.1$ and $\tau=0.07$ (Zhang et al., 10 Jul 2026). This couples language-model continued pretraining to a causal prediction problem defined directly in the latent space of rendered pages.

MVP alters masked image modeling by replacing the tokenizer with the vision branch of CLIP. Instead of reconstructing pixels or d-VAE codes, the model aligns masked-patch and CLS representations to frozen CLIP visual embeddings with a cosine-similarity objective (Wei et al., 2022). This changes the representational target from low-level appearance to a language-grounded semantic space. ViTP shifts the proxy task further: its primary signal is instruction following inside a vision-LLM, augmented with Visual Robustness Learning, which randomly drops a fraction $r$ of visual tokens before the LLM processes them. The VRL loss is another instruction-following negative log-likelihood, but under sparse visual input, forcing the visual encoder to preserve domain-relevant information under token subsampling (Li et al., 22 Sep 2025).

These differences in target space are consequential. A plausible implication is that VP methods inherit the semantics of the space they predict: CLIP-guided MIM inherits multimodal semantics, video reconstruction inherits temporal persistence, and annotation autoregression inherits compositional structure over locations and shapes.

3. Architectural patterns and tokenization strategies

Architecturally, VP spans single-stream, dual-stream, and hybrid encoder-decoder designs. The survey divides vision-language pre-training into single-stream models that concatenate visual and textual tokens into one Transformer, dual-stream models with separate vision and text towers, and hybrid or encoder-decoder systems tailored to generative tasks (Chen et al., 2022). SemVLP is a compact example of architectural unification: it shares a single bidirectional Transformer backbone and activates a pluggable cross-modal attention module only in the higher layers for two-stream operation. Its pretraining alternates, with $50\%$ probability, between single-stream and two-stream modes, thereby aligning image and text at both fine-grained and high-level semantic levels (Li et al., 2021).

Tokenization is equally variable. VisorGPT discretizes continuous coordinates into $m=512$ equally spaced bins after resizing the long edge of images to $512$ px, then wraps the result in prompt templates and applies BPE to the resulting text sequence. The decoder-only Transformer is GPT-2 base style, with 12 self-attention layers, hidden size $768$, 12 heads, a training context window of 1,024 tokens, and approximately 117 M learnable parameters (Xie et al., 2023). This is a notable inversion of conventional visual tokenization: boxes, keypoints, and masks are treated as language-like sequences.

In IV-CL, each video frame is processed by a ViT-Base image encoder with learnable slot tokens prepended to the patch tokens. These slots compress frame information into a small fixed set of vectors that are later consumed by a temporal Transformer and a ViT-style masked-autoencoder decoder (Sun et al., 2023). In VITO, the architecture is closer to contrastive image SSL, but with multi-scale attention pooling applied to features from sampled video frames, and with an EMA target network in a BYOL/MoCLR-style setup (Parthasarathy et al., 2022). The document-centric VP pipeline introduces yet another tokenization: each rendered page is passed through a frozen ViT-style vision tower and spatial merger, then foreground filtering removes blank regions, and the retained visual tokens are projected into the LLM hidden space in raster order (Zhang et al., 10 Jul 2026).

The recurrence of patch tokens, slot tokens, region tokens, and discretized annotation tokens indicates that VP is often determined less by the backbone family than by the granularity at which visual structure is serialized and predicted.

4. Temporal structure, video pretraining, and reasoning

One major research thread argues that static image pretraining underutilizes a key source of structure: temporal evolution. IV-CL addresses end-to-end visual reasoning with a self-supervised framework that compresses each video frame into a small set of slot tokens and reconstructs masked patches from temporal context. On CATER and ACRE, pretraining is reported as essential for compositional generalization: IV-CL achieves $$L=\lambda_{\text{text}}L_{CE}+\lambda_{\text{vis}}L_{VP},$$0 on CATER and $$L=\lambda_{\text{text}}L_{CE}+\lambda_{\text{vis}}L_{VP},$$1 on ACRE compositionality, exceeding supervised pretraining baselines based on classification or explicit object detection by large margins (Sun et al., 2023). The study explicitly frames this as evidence against the view that explicit visual abstraction is always necessary for visual reasoning.

VITO makes a related but broader claim about representation quality. It curates VideoNet, containing approximately 1.18 M clips, and trains a contrastive video objective over augmented frames sampled from the same clip. The resulting representations achieve $$L=\lambda_{\text{text}}L_{CE}+\lambda_{\text{vis}}L_{VP},$$2 mIoU on ADE20K, $$L=\lambda_{\text{text}}L_{CE}+\lambda_{\text{vis}}L_{VP},$$3 on DAVIS’17 $$L=\lambda_{\text{text}}L_{CE}+\lambda_{\text{vis}}L_{VP},$$4 mean, and $$L=\lambda_{\text{text}}L_{CE}+\lambda_{\text{vis}}L_{VP},$$5 top-1 on UCF-101 with multi-scale attention pooling (Parthasarathy et al., 2022). The same paper also emphasizes robustness and human alignment: on ImageNet-A, VITO attains $$L=\lambda_{\text{text}}L_{CE}+\lambda_{\text{vis}}L_{VP},$$6, and on ClickMe saliency alignment it reaches $$L=\lambda_{\text{text}}L_{CE}+\lambda_{\text{vis}}L_{VP},$$7, surpassing several image-pretrained or explicitly saliency-oriented baselines (Parthasarathy et al., 2022).

A more recent extension studies video diffusion pretraining as a route to “visual foundation models.” Using CogVideoX1.5-5B with LoRA adapters, the study reports higher data efficiency than adapted LLMs across ARC-AGI, ConceptARC, visual games, route planning, and cellular automata. Representative numbers include $$L=\lambda_{\text{text}}L_{CE}+\lambda_{\text{vis}}L_{VP},$$8 versus $$L=\lambda_{\text{text}}L_{CE}+\lambda_{\text{vis}}L_{VP},$$9 on ARC-AGI for CogVideoX1.5-5B and Qwen-4B-Instruct, $\lambda_{\text{vis}}$0 accuracy at $\lambda_{\text{vis}}$1 on Sudoku for the VDM while the LLM remains under $\lambda_{\text{vis}}$2, and Maze Base accuracy of $\lambda_{\text{vis}}$3 by $\lambda_{\text{vis}}$4 for the VDM versus roughly $\lambda_{\text{vis}}$5 for the LLM at the same training size (Acuaviva et al., 28 Oct 2025). This suggests that spatiotemporal pretraining can supply inductive biases for structured visual problem solving that are not easily recovered from text-only pretraining.

5. Explicit priors, multimodal guidance, and language intelligence

A distinct strand of VP focuses on making latent visual regularities explicit. VisorGPT is built on the observation that models such as Stable Diffusion and ControlNet learn object location, size, shape, and co-occurrence only implicitly. By learning the joint distribution of discretized boxes, poses, and masks, it yields a controllable prior over plausible scene layouts. On approximately 4 M training sequences, it reports near-perfect sequence validity and matching rates, size-accuracy of approximately $\lambda_{\text{vis}}$6–$\lambda_{\text{vis}}$7, instance-count accuracy of approximately $\lambda_{\text{vis}}$8–$\lambda_{\text{vis}}$9, and COCO prior KL divergences of approximately $0.1$0 for location, $0.1$1 for shape, and $0.1$2 for relation (Xie et al., 2023). The model can then sample layouts for downstream generators such as ControlNet and GLIGEN.

MVP demonstrates that multimodal supervision can strengthen ostensibly image-only pretraining. By using the frozen CLIP vision encoder as the target space for masked image modeling, MVP reports $0.1$3 ImageNet-1K top-1 for ViT-B/16 after 300 epochs, $0.1$4 linear-probe accuracy for ViT-B/16, and $0.1$5 mIoU on ADE20K, surpassing BEIT by $0.1$6 mIoU on that benchmark (Wei et al., 2022). The ablation comparing CLIP and DINO targets shows $0.1$7 versus $0.1$8 for classification and $0.1$9 versus $\tau=0.07$0 for segmentation, indicating that the semantic structure inherited from CLIP’s multimodal training is especially valuable for dense prediction (Wei et al., 2022).

The document-centric VP formulation extends this logic from visual recognition to language intelligence. Instead of parsing scientific PDFs into text, it renders each page as an image and trains the LLM to predict next visual latents in parallel with ordinary text next-token prediction. Across four backbones, VP improves GPQA by up to $\tau=0.07$1 points and MMLU-Pro by up to $\tau=0.07$2, while also raising AIME and HLE modestly, despite using only one-quarter of the token budget on the PDF corpus (Zhang et al., 10 Jul 2026). On multimodal benchmarks for Qwen 3.5, the same work reports improvements from $\tau=0.07$3 to $\tau=0.07$4 on MMMU-Pro, from $\tau=0.07$5 to $\tau=0.07$6 on SFE, from $\tau=0.07$7 to $\tau=0.07$8 on ChartQAPro, and from $\tau=0.07$9 to $r$0 on MathVista, alongside tighter visual-textual alignment as mean cosine similarity rises from $r$1 to $r$2 (Zhang et al., 10 Jul 2026). The underlying claim is not merely that vision helps vision, but that page layout, equations, tables, and figure-caption relations can improve language-intensive reasoning when preserved as visual structure.

ViTP pushes supervision in the opposite direction: rather than using language to annotate a fixed visual encoder, it uses instruction-following gradients to reshape the encoder itself. Initialized from InternVL-2.5 and trained end-to-end on domain-specific instruction corpora, ViTP reports remote-sensing results such as $r$3 mAP on DIOR-R, $r$4 on DOTA-v2.0, and $r$5 mIoU on iSAID, as well as medical results including $r$6 mDice on AMOS2022 and $r$7 on BraTS2021 (Li et al., 22 Sep 2025). Its Visual Robustness Learning ablation peaks at a token-drop ratio of $r$8, with RSAR mAP improving from $r$9 without VRL to $50\%$0 with VRL (Li et al., 22 Sep 2025). This suggests a top-down form of visual pretraining in which reasoning signals actively sculpt perceptual features.

6. Transfer, evaluation, limitations, and open questions

VP is typically justified by transfer, but the transfer protocol itself varies substantially. Some works fine-tune the entire model; others freeze most parameters. PVP shows that pretraining can be applied not only to a backbone but also to the lightweight modules used in parameter-efficient tuning. It pretrains prompt modules on ImageNet-1k while keeping the ViT backbone frozen, then reuses those modules for few-shot downstream tasks. On five FGVC datasets, average 1-shot accuracy rises from $50\%$1 for shallow VPT to $50\%$2 for PVP(VPT), with further gains at 2, 4, and 8 shots (Song et al., 2023). This result broadens the meaning of pretraining: transferable visual structure can reside in adaptation parameters as well as in the core encoder.

Evaluation protocols reflect the heterogeneity of VP. VisorGPT uses sequence validity, matching rate, controllability, and KL divergence to assess the fidelity of learned priors (Xie et al., 2023). IV-CL evaluates compositional reasoning on CATER and ACRE (Sun et al., 2023). VITO emphasizes downstream recognition, robustness to natural and synthetic deformations, and consistency with human judgments (Parthasarathy et al., 2022). Vision-language work evaluates captioning, retrieval, VQA, NLVR, and related tasks (Chen et al., 2022). Document-centric VP measures scientific reasoning, cross-modal alignment, and multimodal transfer (Zhang et al., 10 Jul 2026). The absence of a single canonical benchmark is itself informative: VP is assessed by the properties it is supposed to transfer.

Several limitations recur. VisorGPT is closed-set, limited to approximately 1,000 categories seen during training, and its sequence-length bound of 256–1,024 tokens limits the number of instances or mask points per scene; proposed directions include incorporating open-vocabulary text corpora, extending the context window, and exploring multilingual or zero-shot transfer of visual priors (Xie et al., 2023). MVP depends on an external pre-trained multimodal model, and its frozen CLIP encoder imposes a fixed semantic space (Wei et al., 2022). The document-centric VP study focuses on visually dense scientific PDFs, and explicitly identifies generalization to broader visual corpora such as natural images and video as an open question, alongside more sophisticated foreground selection and loss scheduling (Zhang et al., 10 Jul 2026). More broadly, the survey on VLP highlights challenges around efficiency, compression, prompt-based adaptation, out-of-domain robustness, knowledge grounding, and expansion beyond two modalities (Chen et al., 2022).

Taken together, these works indicate that visual pretraining is not reducible to “pretraining a vision encoder on images.” It is a broader research program concerned with how visual structure should be represented, what proxy task should be optimized, and which invariances should be inherited before downstream supervision. The current literature supports several nonexclusive conclusions: temporal structure can improve reasoning and robustness; multimodal targets can strengthen semantic content; explicit priors over layouts and annotations can make generation more controllable; and preserving the visual form of documents can benefit language intelligence itself.