Large Vision Models Overview

• Large Vision Models (LVMs) are high-capacity visual foundation models that leverage large pretrained backbones and extensive training data to transfer across diverse tasks.
• They employ varied architectures, including autoregressive tokenization, encoder-decoder frameworks, and sparse expert modalities to handle tasks such as segmentation, reconstruction, and cross-modal reasoning.
• LVMs drive advancements in fields from autonomous driving to medical diagnostics by improving efficiency, robustness, and performance in both vision-only and multimodal applications.

Large Vision Models (LVMs) are a class of high-capacity visual foundation models defined in recent literature by scale, pretraining, transferability, and flexible downstream adaptation. Across the cited work, the term is used in two closely related senses. In a narrower sense, it denotes vision-only systems trained directly on images, videos, 3D inputs, or other visualized signals without linguistic supervision; in a broader sense, especially in domain benchmarks and hallucination studies, it overlaps with large vision-LLMs (LVLMs) and large language and vision models (LLVMs), where a vision encoder is coupled to a projector and an LLM for multimodal reasoning or generation (Bai et al., 2023, Xu et al., 1 Aug 2025, Lee et al., 2024). The common research program is to replace narrow task-specific pipelines with large pretrained visual backbones that can be scaled, prompted, fine-tuned, or structurally adapted across recognition, generation, reconstruction, and cross-modal reasoning tasks (Jin et al., 2024, Wu et al., 3 Mar 2026, Zhang et al., 2024).

1. Terminological scope and conceptual definition

In the vision-only literature, LVMs are explicitly framed as the vision-side analogue of LLMs: scale plus sequential modeling plus prompting, but without linguistic data. “Sequential Modeling Enables Scalable Learning for Large Vision Models” defines an LVM as an autoregressive Transformer trained over tokenized “visual sentences,” where raw images, videos, and visually rendered annotations are all converted into a shared sequence format and optimized with a GPT-style next-token objective (Bai et al., 2023). In application-oriented surveys, LVMs are defined more generally as large pretrained vision foundation models—typically based on Transformers, GANs, autoencoders, or diffusion models—that learn reusable visual representations for classification, detection, segmentation, reconstruction, generation, and multimodal processing (Xu et al., 1 Aug 2025).

The nomenclature is not fully uniform. In human-robot interaction, LVMs are described as “broad, pretrained, high-capacity models” such as CLIP, ViT, Swin Transformer, GPT-4V, and SAM, with emphasis on domain alignment rather than a single architecture family (Zhang et al., 2024). In medical and reliability-focused work, the same conceptual space is often discussed under the label LVLM: models that combine a vision encoder with a LLM to interpret images and generate text, as in ophthalmology benchmarks or hallucination mitigation studies (Qin et al., 2024, Zhang et al., 31 Jan 2025). A standard LLVM formalization writes the multimodal model as

$$\mathcal{M} = f_\text{L} \circ f_\text{P} \circ f_\text{V},$$

where $f_\text{V}$ is a pretrained vision encoder, $f_\text{P}$ is a projector, and $f_\text{L}$ is a pretrained LLM (Lee et al., 2024).

A stable conceptual core nevertheless emerges. LVMs are large because they use large backbones, large training corpora, or both; they are “foundation” models because their representations are intended to transfer across tasks and domains; and they are increasingly evaluated not only by closed-task accuracy, but also by zero-shot behavior, prompting, cross-domain robustness, and efficiency under adaptation (Bai et al., 2023, Xu et al., 1 Aug 2025).

2. Architectural paradigms and representational strategies

One major architectural line treats vision as a pure sequence domain. In the visual-sentence framework, each $256 \times 256$ image is tokenized into 256 discrete tokens using a VQGAN-style tokenizer with downsampling factor $f=16$ and codebook size 8192, and a decoder-only LLaMA-like Transformer is trained to model

$$p(x_t \mid x_{<t}), \qquad \mathcal{L} = - \sum_t \log p(x_t \mid x_{<t}).$$

This representation unifies raw images, videos, semantic segmentations, depth reconstructions, surface normals, edges, pose renderings, stereo pairs, and other annotated outputs as a single autoregressive token stream (Bai et al., 2023).

A second line removes explicit geometric structure from 3D-aware generation. LVSM, described as a “Large View Synthesis Model,” addresses novel view synthesis from sparse posed images using a fully transformer-based token-to-token mapping with minimal 3D inductive bias. Input images are paired with Plücker ray embeddings, patchified, linearly projected into tokens, and decoded into target-view patches. The encoder-decoder variant compresses inputs into a fixed set of 1D latent tokens, whereas the decoder-only variant eliminates the latent bottleneck entirely and performs direct conditional generation. Both variants explicitly avoid NeRF-style volumetric fields, 3D Gaussian Splatting, tri-plane NeRFs, meshes, voxels, point-based renderable representations, epipolar projections, plane-sweep volumes, cost volumes, and related handcrafted geometric operators (Jin et al., 2024).

Unified multimodal LVMs introduce a third paradigm. PolyV argues that functional integration across image, video, and 3D inputs is insufficient unless the model can exploit complementary priors bidirectionally. Its backbone replaces standard FFNs with sparse MoE layers coordinated by a dynamic modality router; the implementation uses 4 experts per MoE layer, activates top-2 experts per token, and interleaves MoE layers every 4 layers. This design is meant to let tokens from one modality be processed by experts that have specialized on other modalities, yielding “cross-vision synergy” rather than shallow fusion (Wu et al., 3 Mar 2026).

In multimodal vision-language systems, the dominant pattern remains the stitched architecture of frozen or partially frozen visual encoders plus lightweight adapters and LLM decoders. RelationVLM follows this template with a BLIP-2-pretrained ViT/Q-Former encoder, a trainable linear adapter, and Vicuna-7B, then specializes the training pipeline toward semantic relations, temporal associations, and geometric transforms across single images, multiple images, and videos (Huang et al., 2024).

3. Training, scaling, and efficient adaptation

Scaling laws and training scale are central to the LVM literature. The visual-sentence LVM is trained on the Unified Vision Dataset v1, comprising 1.64 billion images/frames and roughly 420 billion visual tokens, with model scales of 300M, 600M, 1B, and 3B parameters. Its largest model uses hidden dim 3200, MLP dim 8640, 32 heads, 26 layers, context length 4096, and trains for about 14 days on a v3-512 TPU pod. Larger models reduce training loss faster, and the reduction in perplexity transfers to downstream visual prompting tasks such as semantic segmentation, depth estimation, surface normal estimation, and edge detection (Bai et al., 2023).

LVSM provides a second scaling example in a geometry-heavy domain. Its default implementation uses patch size $p=8$, token dimension $d=768$, and 24 transformer layers; the encoder-decoder version uses 12 encoder plus 12 decoder layers, together with QK-Norm, FlashAttention-v2, gradient checkpointing, and bf16. Final models were trained on 64 A100 GPUs for 3–7 days, but the paper also emphasizes that scaled-down variants trained on only 1–2 A100 GPUs still outperform prior methods; a smaller decoder-only model trained on a single A100 for 7 days reaches 27.66 dB PSNR, beating the best prior 1-GPU baseline by 1.3 dB (Jin et al., 2024).

Because the cost of training and fine-tuning LVMs is itself a major research problem, several works focus on structural efficiency. AutoProg introduces automated progressive learning for ViTs and diffusion models. AutoProg-One combines momentum growth with a one-shot growth schedule search for pre-training, while AutoProg-Zero uses zero-shot unfreezing schedule search and a Unique Stage Identifier for transfer learning and fine-tuning. The reported speedups reach up to 1.85x for ViT pre-training on ImageNet and up to 2.86x for diffusion-model fine-tuning, with comparable or even higher performance (Li et al., 2024).

A complementary efficiency result appears in layer-selective LVLM tuning. “Activating Distributed Visual Region within LLMs” reports that selectively updating 25% of LLM layers, when sparsely and uniformly distributed, can preserve nearly 99% of visual performance while reducing training time. On Bunny-Llama-3-8B-V, the full-tuning average across eight visual benchmarks is 63.51, while sparse uniform 8-layer tuning reaches 62.88, or 99.00% of full performance; the same work reports about 23% reduction in training time for LLaVA-1.5-7B and LLaVA-1.5-13B, and about 12–13% for Bunny-Llama-3-8B-V (Wang et al., 2024).

4. Task breadth and domain-specific deployment

The empirical range of LVM applications is unusually broad. In human-robot interaction, LVMs are organized through a three-dimensional design space of HRI contexts, vision-based tasks, and specific domains. The task axis includes detection, recognition, segmentation, tracking, classification, scene reconstruction, pose estimation, captioning, and visual information generation, while the domain axis spans healthcare, automotive, manufacturing, entertainment, security, agriculture, education, and social interaction. The same work argues that domain-specific LVMs outperform general ones in targeted HRI scenarios and states that such models require only 10% to 30% of the labeled data and produce significantly fewer errors (Zhang et al., 2024).

Large vision models are also being exported into non-image scientific data by visual transformation. A principled time-series study evaluates 4 LVMs, 8 imaging methods, 18 datasets, and 26 baselines, concluding that LVMs are useful for time series classification but much less reliable for forecasting. ViT and MAE are the strongest LVM families; GAF is the best imaging method for classification, while UVH and MVH are best for forecasting because they preserve raw values in a reconstruction-compatible form (Zhao et al., 29 May 2025). A more specialized long-term forecasting framework, DMMV, then integrates numerical and visual views of the same series through decomposition and gating; it reports the best mean squared error on 6 out of 8 benchmark datasets and is explicitly designed to exploit the inductive bias of visual forecasters toward periodic structure (Shen et al., 29 May 2025).

Wireless communications and IoT provide another transfer domain. Survey work characterizes LVMs as a backbone for channel modeling, beamforming, modulation recognition, routing, anomaly detection, semantic communications, localization, and wireless digital twins (Xu et al., 1 Aug 2025). LVM4CSI makes this transfer especially explicit: it maps channel estimation to object detection, human activity recognition to image classification, and user localization to feature extraction plus regression, while keeping the pre-trained LVM frozen and learning only lightweight heads where necessary. The reported gains include an improvement exceeding 9.61 dB in channel estimation and approximately 40% reduction in localization error (Guo et al., 7 Jul 2025).

Autonomous driving has motivated both centralized and federated LVM deployment. One line uses pretrained ImageGPT as a backbone for street-scene semantic understanding and introduces Posterior Optimization Trajectory-Guided optimization to accelerate convergence; the method is reported to converge within 10 epochs and to improve performance by over 66.48% relative mIoU on CamVid versus plain LVM (Kou et al., 3 Jan 2025). A second line, pFedLVM, places the LVM only on the central server in federated autonomous driving, exchanges latent features rather than LVM parameters, and combines shared features with vehicle-specific characteristics to achieve personalized learning; on CamVid it reports improvements of 18.47% mIoU, 25.60% mF1, 51.03% mPrecision, and 14.19% mRecall over the compared state-of-the-art approach (Kou et al., 2024).

Vision-only foundation features are also being used to address long-tailed recognition without language. A SAM-based method augments a ResNet-50 backbone through map-space and latent-space fusion, then adds prototype-based losses tailored separately to head and tail classes. On ImageNet-LT, CE improves from 41.6 overall to 46.9 with SAM and prototype loss, while Label Shift improves from 48.2 to 52.4 and ProCo from 57.2 to 58.8 (Han et al., 15 Apr 2025).

5. Empirical behavior, benchmarks, and limitations

Performance claims for LVMs are often strong, but benchmark behavior is highly task- and domain-dependent. In novel view synthesis, both LVSM variants are reported to achieve state-of-the-art quality, with the decoder-only version outperforming previous methods by 1.5 to 3.5 dB PSNR overall, including gains of about 3 dB on ABO, 2.8 dB on GSO, and about 1.6 dB on RealEstate10K relative to GS-LRM; the same model generalizes zero-shot to different numbers of input views and can often work with a single input image (Jin et al., 2024). In relation-centric multimodal reasoning, RelationVLM reports 49.3 BBox Acc and RS = 2.5 on semantic similarity, 83.2% accuracy and RS = 3.3 on semantic contrast, 69.7 BBox Acc and RS = 3.2 on geometric transformation, and 72.0% on MVTec AD plus 65.3% on ISIC in 8-shot settings (Huang et al., 2024).

At the same time, domain-specific evaluation can expose large performance gaps. LMOD, a benchmark of 21,993 ophthalmology instances across OCT, CFP, SLO, lens photographs, and surgical scenes, reports an average anatomical recognition F1 of 0.2189 across 13 LVLMs, with GPT-4o as best at F1 = 0.5748; diagnosis performance is also weak, with average glaucoma accuracy 50.31% and average macular hole staging accuracy 15.31%. By contrast, supervised baselines trained directly on the tasks achieve 94.36 macro-average F1 for anatomical recognition, 82.69% glaucoma accuracy, and 98.17% macular hole staging accuracy. The benchmark also identifies six failure modes: misclassification, failure to abstain, inconsistent reasoning, hallucination, assertions without justification, and lack of domain-specific knowledge (Qin et al., 2024).

A separate line of analysis examines what current large language and vision models are actually doing internally. “Intriguing Properties of Large Language and Vision Models” reports that random permutation of visual patch sequences causes an average drop of only 0.19 points for LLaVA-1.5 across 10 benchmarks, i.e. less than 1%, suggesting a largely global and approximately permutation-invariant use of image tokens. The same study finds that visual instruction tuning can substantially reduce the original encoder’s zero-shot classification performance, with average drops of 29.2 points for LLaVA-1.5 relative to CLIP ViT 336 and 22.1 points for LLaVA-NeXT relative to CLIP ViT 14, and concludes that the lower layers, specifically the lower part of the stack with emphasis on the first quarter, are crucial for performance and visual understanding (Lee et al., 2024).

Time-series transfer studies reveal a different kind of limitation. Although pretraining helps and self-supervised LVMs such as MAE outperform supervised variants for forecasting, the best current LVM forecasters remain biased toward “forecasting periods,” improve only up to a point with longer look-back windows, and can plateau or decline when the input history becomes too long for a fixed image representation (Zhao et al., 29 May 2025).

6. Robustness, concept selection, and emerging research directions

Current work is increasingly organized around three corrective themes: better grounding, more efficient visual abstraction, and richer multimodal interaction. The most direct grounding intervention is VAP, a training-free visual adversarial perturbation method for object hallucination in LVMs. VAP formulates hallucination suppression as an optimization problem over small visual perturbations, uses zeroth-order optimization, leaves the base model unchanged, and reports consistent improvements across 8 state-of-the-art LVMs on POPE, BEAF, and CHAIR. The paper emphasizes that Gaussian noise of the same strength harms performance, whereas optimized perturbations improve factual grounding, and that VAP is compatible with model-centric methods such as visual contrastive decoding (Zhang et al., 31 Jan 2025).

A second direction seeks to move beyond dense token processing. VCM introduces concept-level visual reasoning through implicit contrastive learning, dynamic visual token selection, and segment merging. Rather than forwarding all image tokens into the LLM, VCM learns sparse instruction-relevant visual concepts $\mathbf{H}^\text{V}_C$ from dense visual tokens $f_\text{V}$0, then optimizes a combined next-token prediction and concept-modeling loss. On LLaVA-1.5-7B, it reports about 85% fewer FLOPs while retaining 98.6% of the original performance with only 64 visual tokens, and it also improves zero-shot classification, open-vocabulary detection, and open-vocabulary semantic segmentation of the visual encoder itself (Luo et al., 28 Apr 2025).

A third direction is explicit cross-modal or cross-vision coordination. PolyV argues that a unified LVM should do more than accept multiple input types; it should reason over complementary priors from image, video, and 3D data through sparse expert specialization and synergy-aware training. Initialized from Qwen2.5-VL-7B, trained with modality-specific pretraining plus coarse-to-fine synergy tuning, and augmented with a CSQA dataset of 20K QA pairs, PolyV reports over 10% average improvement over its backbone across 10 benchmarks and especially large gains on synergy-focused evaluations such as CV-Bench, DSI-Bench, and STI-Bench (Wu et al., 3 Mar 2026).

This trajectory suggests that the central unresolved problem is not merely scaling parameter count. The literature increasingly points to a more specific agenda: preserving perceptual competence under cross-modal alignment, selecting only task-relevant visual concepts instead of forwarding dense token streams, and exploiting complementary priors across modalities and domains without collapsing them into a single undifferentiated representation (Lee et al., 2024, Qin et al., 2024).