Visual Pretraining Overview
- Visual pretraining is a methodology that trains vision models on large-scale, generic data to learn transferable features across various tasks.
- It employs techniques such as contrastive learning, masked image modeling, and generative captioning to enhance performance in classification, detection, and reasoning.
- Recent advances integrate language and domain-specific supervision, bridging the gap between visual representation and multimodal reasoning.
Visual pretraining is the stage where a vision model is trained on large generic data before being adapted to specific tasks, and, in vision-and-language settings, also denotes how the visual part of a model is initialized from a strong vision backbone and further trained jointly with text to produce visual representations that are compatible and aligned with language (Qian, 2023, Long et al., 2022). Across recent work, the term spans supervised image-level pretraining, self-supervised contrastive and masked-image objectives, caption-based and location-aware generative schemes, object-centric and multimodal contrastive methods, domain-specific knowledge-enhanced training, reinforcement-learning formulations, and document-native visual pretraining aimed at language intelligence rather than conventional image understanding alone (Yang et al., 2023, Wan et al., 2024, Ghosh et al., 13 Jun 2025, Zhang et al., 10 Jul 2026).
1. Definitions, scope, and historical broadening
Early visual pretraining in vision-and-language pretrained models commonly started from region-of-interest features extracted by Faster R-CNN and represented an image as a BERT-style sequence of visual tokens, each combining visual content, spatial position, and modality embeddings (Long et al., 2022). In parallel, foundation vision models were commonly pretrained either with supervised classification on ImageNet-style labels or with self-supervised objectives such as contrastive learning and masked image modeling, with the aim of producing a general-purpose encoder reusable for classification, detection, and segmentation (Qian, 2023).
Subsequent work expanded this scope along three axes. First, token granularity diversified from RoI features to grid, patch, and pixel-level representations, including ViT-style patch embeddings and end-to-end raw-pixel vision-LLMs (Long et al., 2022, Yang et al., 2023). Second, supervision diversified from curated labels to captions, weak metadata, detector pseudo-labels, prototypes, and self-supervised signals, so that visual pretraining no longer implied a single annotation regime (Sariyildiz et al., 2020, Yang et al., 2023, Beal et al., 2021). Third, the target of pretraining diversified: some methods optimized transferable visual representations for standard CV tasks, whereas others optimized multimodal grounding, document understanding, or language reasoning over visually rich corpora (Zhou et al., 2024, Zhang et al., 10 Jul 2026).
A recurrent misconception in older multimodal pipelines was that vision mainly served as conditioning context for text generation or retrieval. More recent work argues for a broader role: visual pretraining can shape the predictive process itself, including dense region-text alignment, document-layout reasoning, and latent-space objectives that alter downstream language capabilities (Shi et al., 2021, Zhang et al., 10 Jul 2026). This suggests that “visual pretraining” now refers less to a single recipe than to a family of pretraining regimes in which visual structure is the carrier of transferable inductive bias.
2. Objective families and supervisory signals
The literature in the provided corpus organizes visual pretraining around several recurring objective families.
| Paradigm | Representative objective or mechanism | Representative papers |
|---|---|---|
| Supervised and multi-task visual pretraining | Multi-label classification combined with MIM, contrastive clustering, and momentum distillation | (Qian, 2023) |
| End-to-end semantic visual losses | Self-supervised patch prediction, semantic segmentation loss, and set prediction loss on raw pixels | (Yang et al., 2023) |
| Caption-grounded visual pretraining | Image-conditioned masked language modeling and caption-derived tag prediction | (Sariyildiz et al., 2020) |
| Object-centric contrastive learning | Region-level InfoNCE on mask-pooled features across augmented views | (Hénaff et al., 2021) |
| Location-aware generative pretraining | Autoregressive generation of captions and bounding-box coordinates in one token stream | (Wan et al., 2024) |
| Dense visual-linguistic contrast | Region-level contrastive learning replacing feature regression and label classification | (Shi et al., 2020, Shi et al., 2021) |
| Visual prior generation | GPT-style likelihood over discretized boxes, poses, and masks | (Xie et al., 2023) |
| Document-native visual pretraining | Next visual latent prediction jointly with text loss | (Zhang et al., 10 Jul 2026) |
| RL-based visual pretraining | Bellman-style value learning over view transformations and annotation rewards | (Ghosh et al., 13 Jun 2025) |
One influential line combines supervised and self-supervised signals. In HVP-MTL, the total loss is
where multi-label classification supplies semantic supervision, MIM reconstructs masked pixels, contrastive clustering promotes view-invariant structure, and momentum distillation mitigates label noise (Qian, 2023). This formulation directly encodes the view that labels and self-supervision are complementary rather than mutually exclusive.
A second line replaces detector-centric supervision with contrastive objectives. CVLP removes supervised region regression and classification in favor of a visual self-supervised contrastive loss built on QueryNet, KeyNet, a momentum encoder, and a memory queue, while retaining masked language modeling and image-sentence matching (Shi et al., 2020). DCVLP pushes the same idea toward dense cross-modality region contrastive learning and adds mask perturbation and intra-/inter-adversarial perturbation to improve negative samples in a fully annotation-free visual branch (Shi et al., 2021).
A third line uses language-like generation as the supervisory interface. ICMLM predicts masked caption tokens conditioned on image features, thereby forcing the visual encoder to encode caption semantics and localized concepts (Sariyildiz et al., 2020). LocCa treats coordinates as ordinary tokens and trains one encoder-decoder model on captioning, referring expressions, and grounded captioning, so that location becomes part of the same autoregressive stream as text (Wan et al., 2024). VisorGPT extends generative pretraining to discretized visual annotations themselves, modeling boxes, keypoints, and masks with a GPT-style likelihood objective rather than pixels or contrastive pairs (Xie et al., 2023).
The most explicit departure from conventional SSL appears in annotation bootstrapping. There, the reward model can be CLIP-, SimCLR-, or DINO-like, but the core pretraining step learns a value function over view transformations with a Bellman target, turning crop-consistency into a general value-function problem rather than a direct invariance constraint (Ghosh et al., 13 Jun 2025). A plausible implication is that visual pretraining objectives can be organized not only by supervision source, but also by whether they learn invariance, equivariance, reconstruction, generation, or long-horizon semantic propagation.
3. Representations and architectures
Visual pretraining methods differ sharply in their choice of visual tokens. RoI-based VLPMs represent each detected region with a projected region feature and normalized box coordinates, while patch-based and ViT-based systems treat non-overlapping image patches as tokens and rely on learned or sinusoidal positional embeddings (Long et al., 2022). Region-based designs retain strong object priors but inherit detector bias; patch and pixel designs are more end-to-end but must learn semantics directly from raw visual input.
End-to-end vision-language pretraining from raw pixels is difficult precisely because pixel features lack the semantic priors that region features provide. One response is to add auxiliary visual losses. In the semantic-visual-loss framework, the model uses a visual encoder from raw pixels, a language encoder, and a cross-modality encoder, then augments standard MLM and ITM with SSUL, SEGL, or SPL. SEGL applies a low-resolution semantic segmentation cross-entropy, while SPL distills object labels from an offline detector via Hungarian matching, reducing visual tokens and preserving end-to-end inference at deployment time (Yang et al., 2023).
Other systems force object-centric structure more directly. DetCon uses unsupervised segmentation masks, masked average pooling over a convolutional feature grid, and an InfoNCE-style loss over region features across two augmented views; with a single global mask, the objective collapses back to SimCLR or BYOL-style instance discrimination (Hénaff et al., 2021). LocCa instead keeps a unified encoder-decoder architecture but injects localization by having the decoder emit coordinate strings in the same SentencePiece vocabulary as natural language, which makes object location part of the sequence model itself (Wan et al., 2024).
Multimodal architectures also vary in fusion strategy. The survey distinguishes self-attention-based single-stream fusion, co-attention-based two-stream models, and dual-encoder visual-semantic embedding models (Long et al., 2022). CVLP and DCVLP are transformer-based visual-linguistic systems that keep separate visual and language encoders but perform multimodal fusion before contrastive region supervision (Shi et al., 2020, Shi et al., 2021). ViCHA adds a further architectural twist by concatenating CLIP-derived visual concept tokens to image patch tokens before the ViT encoder, effectively injecting semantic prompts into the visual stream (Shukor et al., 2022).
Document-native visual pretraining introduces a different representation problem. In VP for language intelligence, each scientific PDF page is rendered as an RGB image, encoded by a frozen ViT-style vision tower, spatially merged into coarser visual latents, foreground-filtered to remove whitespace, and rasterized into a sparse sequence of foreground visual tokens:
These projected visual latents are processed by the same autoregressive backbone used for language, and training predicts the next visual latent under causal masking rather than discrete text tokens (Zhang et al., 10 Jul 2026). The representation is compressed, layout-preserving, and explicitly designed for document structure rather than natural-image semantics.
4. Multimodal, cross-lingual, domain-specific, and language-centered variants
A major branch of visual pretraining is explicitly multimodal. VTLM extends Translation Language Modeling with visual region features and masked region classification on image–English–German triplets, producing a single-stream transformer over source tokens, target tokens, and visual region tokens (Caglayan et al., 2021). In this setting, visual pretraining is not just alignment with one language but a bridge across languages, where images provide a shared referent for multilingual representations. The paper reports that VTLM-based multimodal machine translation establishes new state of the art on Multi30k and that multimodal fine-tuning is only effective when visual grounding is already present in pretraining (Caglayan et al., 2021).
In domain-specialized settings, the same logic appears with different knowledge sources. Knowledge-enhanced visual-language pretraining for computational pathology builds PathKT with 4,718 diseases, 32 tissues, and 50,470 attributes, trains a PubMedBERT-based knowledge encoder with AdaSP metric learning, then regularizes CLIP-style image-text training with a frozen knowledge branch and a text–knowledge contrastive loss (Zhou et al., 2024). This modifies the text side of pretraining so that image features align not only with noisy captions but also with a disease-structured semantic space.
ViCHA, although not domain-specific, also argues that visual pretraining quality can be increased by injecting semantic structure directly into the vision encoder. Its training objective combines ITM, MLM, hierarchical image–text contrastive alignment across multiple transformer layers, and MAE-style U-MIM:
with visual concepts extracted by CLIP and injected as additional tokens (Shukor et al., 2022). Here, visual pretraining is partly delegated to an external foundation model, but the downstream encoder is still trained to internalize those concepts.
The most radical language-centered formulation in the corpus is document-native VP. It augments standard text pretraining with a next-visual-latent loss:
where is an InfoNCE-type next visual latent prediction loss (Zhang et al., 10 Jul 2026). The resulting system is not a document parser and does not use OCR, LaTeX parsing, or image–text pairs. Instead, it directly trains an autoregressive language backbone on rendered scientific pages. This design explicitly challenges the assumption that scalable language intelligence must be learned from text-only projections of documents.
5. Empirical behavior, transfer, and applications
Empirically, visual pretraining has repeatedly improved transfer across classification, detection, segmentation, retrieval, multimodal reasoning, and language tasks. HVP-MTL reports that with a vanilla Swin-B backbone, the pretrained model achieves 85.3% top-1 accuracy on ImageNet-1K classification, 47.9 box AP on COCO object detection for Mask R-CNN, and 50.6 mIoU on ADE-20K semantic segmentation when using UperNet (Qian, 2023). The same study shows that removing multi-label classification or MIM degrades ImageNet-1K transfer, reinforcing the complementarity of semantic labels and structural self-supervision.
Object-centric contrastive learning shows a similar pattern. DetCon with a ResNet-50 backbone reaches 42.7 box AP and 38.2 mask AP on COCO under ImageNet pretraining, and a larger ResNet-200 2× variant reaches 48.9 box AP and 43.0 mask AP, essentially matching SEER’s 48.5 and 43.2 while using approximately 1000× less pretraining data (Hénaff et al., 2021). The same paper attributes much of the gain to richer supervision per image through many region-level positives and negatives.
For end-to-end multimodal systems, auxiliary visual losses substantially improve both efficiency and accuracy. The semantic-visual-loss framework reports that a ViT + SEGL model reaches 71.69 on VQAv2 test-dev, 72.03 on test-std, 72.3 on NLVR2 val, 72.9 on NLVR2 test-P, and 58.33 on VCR Q→AR, while running at approximately 40 ms inference compared with approximately 900 ms for region-feature models such as ViLBERT, LXMERT, and UNITER (Yang et al., 2023). The paper concludes that semantic visual losses are sufficient to close much of the gap between detector-based and raw-pixel VL pretraining without the detector’s runtime cost.
Location-aware captioning disproportionately benefits localization-heavy tasks. LocCa reports a zero-shot ImageNet-1k score of 77.1% for LocCa-L versus 76.4% for CaPpa-L, COCO retrieval text→image of 46.6% versus 43.9%, and image→text of 64.6% versus 60.3%; it also reports that LocCa-L vs SigLIP-L in PaLI-3 improves COCO from 135.8 to 138.9 CIDEr, VQAv2 from 75.6 to 77.6, TextVQA from 41.1 to 49.2, and ST-VQA from 46.2 to 50.9 (Wan et al., 2024). The pattern is consistent with the paper’s central claim that explicit spatial supervision during pretraining strengthens both region-level and object-centric reasoning.
Large-scale industrial pretraining confirms that scale interacts strongly with architecture. Pinterest’s billion-scale weakly supervised pretraining replaces a ResNeXt backbone with ViT-B variants and reports a 36% improvement in top-1 relevance and 23% improvement in click-through volume in a production visual shopping system (Beal et al., 2021). The study further shows that ViT advantages emerge most clearly at very large data scale and with domain-specific weak supervision.
Document-native VP extends these gains into language-centric evaluation. In the matched-corpus setup, PDFs rendered to visual tokens contribute about 20B visual tokens, compared with about 80B text tokens extracted from the same PDFs, and total continued-pretraining budgets are about 120B tokens for VP versus about 180B for text pretraining (Zhang et al., 10 Jul 2026). Yet for Qwen 3.5 the VP model reports 85.09 on MMLU-Pro, 79.29 on GPQA, 90.21 on AIME, and 16.67 on HLE, compared with 83.91, 76.24, 89.58, and 15.70 under matched text pretraining; on ChartQAPro it reaches 61.80 versus 56.42 for text pretraining (Zhang et al., 10 Jul 2026). The same paper reports improved cross-modal alignment without explicit image–text supervision, with centroid separation improving from 1.665 to 0.661 and mean paired cosine similarity from 0.631 to 0.907 (Zhang et al., 10 Jul 2026).
Finally, pretraining changes what is feasible in end-to-end reasoning. The video-based IV-CL work concludes that pretraining is essential to achieve compositional generalization for end-to-end visual reasoning and that its self-supervised framework outperforms supervised pretraining, including image classification and explicit object detection, on CATER and ACRE (Sun et al., 2023). This does not show that explicit abstractions are never useful; it does show that suitably designed visual pretraining can induce implicit abstractions strong enough for tasks previously thought to require hand-designed symbolic intermediates.
6. Limitations, debates, and likely directions
The literature does not support a single dominant recipe. Region-based methods can benefit from strong object priors, but they inherit detector noise, domain bias, and annotation sparsity; this is precisely the motivation behind CVLP and DCVLP’s move toward label-free contrastive region supervision (Shi et al., 2020, Shi et al., 2021). End-to-end pixel models remove detector dependence but need auxiliary semantic losses or carefully chosen self-supervised objectives to converge efficiently (Yang et al., 2023). Object-centric contrastive methods such as DetCon remain sensitive to the quality of unsupervised masks, and the paper explicitly shows performance tracking mask quality (Hénaff et al., 2021).
Several approaches also reveal nontrivial domain assumptions. Document-native VP is evaluated on high knowledge-density scientific PDFs and is applied as continued pretraining on top of text-pretrained LLMs rather than from scratch; the paper states that it does not show a fully visual-pretrained-from-scratch LLM and that transfer beyond scientific PDFs is unclear (Zhang et al., 10 Jul 2026). Knowledge-enhanced pathology pretraining relies on PathKT coverage and on noisy public pathology captions; its strengths are domain-specific and tied to a curated ontology (Zhou et al., 2024). LocCa relies on pseudo boxes from OWL-ViT and acknowledges that zero-shot multi-box detection remains an open problem (Wan et al., 2024). VisorGPT is currently closed-set within approximately 1,000 categories and is limited by maximum sequence length when many instances must be modeled in one sequence (Xie et al., 2023).
A second debate concerns invariance versus structure. Classical crop-consistency methods and CLIP-style alignment remain strong baselines, but the RL formulation argues that strict invariance is often too weak a target for complex scenes and that value prediction over transformations can better exploit unlabeled data, especially when reward learning is decoupled from bootstrapping on less curated corpora (Ghosh et al., 13 Jun 2025). A plausible implication is that future visual pretraining may increasingly mix reward models, transformation-aware value learning, and multimodal supervision rather than choosing between them.
A third debate concerns whether visual pretraining is mainly for vision. The evidence in the corpus points in the opposite direction. VTLM uses visual tokens to improve multimodal machine translation (Caglayan et al., 2021). Pathology KEP uses structured medical knowledge to improve zero-shot diagnosis-oriented retrieval and classification (Zhou et al., 2024). Document-native VP improves text-only scientific reasoning by training directly on rendered pages instead of textualized documents (Zhang et al., 10 Jul 2026). The field therefore increasingly treats visual pretraining as a general representation-learning strategy for systems that must reason over objects, layouts, pages, trajectories, or other structured visual substrates.
Future directions stated in the surveyed work are correspondingly broad: better fine-grained alignment and multimodal pretraining schedules (Long et al., 2022); improved integration of labels and self-supervision in vision backbones (Qian, 2023); more efficient end-to-end VL objectives and tokenization strategies (Yang et al., 2023); stronger localization-aware pretraining and segmentation-aware generative interfaces (Wan et al., 2024); ontology-aware multimodal pretraining in specialized domains (Zhou et al., 2024); and visually native objectives such as next-latent prediction or highly compressed visual streams for LLMs (Zhang et al., 10 Jul 2026). Taken together, these strands indicate that visual pretraining is evolving from a preparatory stage for image recognition into a heterogeneous set of foundation-model training strategies for perception, grounding, and reasoning.
