Vision Transformers (ViTs) Overview

• ViTs are vision models that divide images into patches, embed them, and process the sequence with transformer encoder blocks to capture long-range dependencies.
• They leverage self-attention mechanisms and specialized tokenization techniques, but may require data augmentation or hybrid strategies on limited datasets.
• ViTs enable flexible compute allocation through methods like pruning and token selection, and are applied in areas from image classification to video and medical imaging.

Vision Transformers (ViTs) are vision models that divide an image into patches, linearly embed the resulting patch vectors, add positional information, and process the token sequence with stacked transformer encoder blocks built from multi-head self-attention and feedforward sublayers. In the formulation summarized in the survey literature, the core attention operator is

$$\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left( \frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}} \right) \mathbf{V},$$

and the tokenized visual representation is completed by positional encoding and, commonly, a classification token (Mia et al., 2023). Across the literature, ViTs are presented as broadly reusable visual backbones with strong global-context modeling, but also as architectures whose behavior depends sharply on inductive bias, data regime, tokenization strategy, and downstream adaptation protocol.

1. Architectural basis and inductive-bias profile

The basic ViT pipeline in the survey literature is: divide the input image into patches, flatten and linearly embed them, add positional encodings, prepend a class token, and feed the sequence into transformer encoder layers comprising multi-head self-attention and MLP blocks (Mia et al., 2023). The same source attributes several of the architecture’s central advantages to self-attention: direct modeling of long-range dependencies, scalability with large pretraining corpora, and reuse of a unified transformer design across classification, detection, segmentation, video understanding, restoration, and multimodal learning (Mia et al., 2023).

The same literature also emphasizes the liabilities of the vanilla design. ViTs are described as lacking the hard-coded local inductive biases of convolution, as being data-hungry on limited datasets, and as imposing substantial training and deployment cost (Mia et al., 2023). The small-data literature makes this tension concrete. In the CIFAR-10 study of tiny models, the patch-token representation is written as

$x_\text{token} = x_e + x_p,$

where $x_e$ is the patch embedding and $x_p$ is the positional embedding; within that setting, learnable positional embeddings outperform fixed sinusoidal encodings, with 93.65% versus 93.43% validation accuracy (Wu, 7 Jan 2025). This suggests that even seemingly minor representational choices at the tokenization stage can materially affect performance when the model scale and dataset size are both constrained.

A recurrent theme across the papers is that ViTs should not be treated as a single monolithic architecture. The survey enumerates plain ViTs, DeiT, Swin, CrossViT, CSWin, DETR-like detection models, SegFormer-style segmentation models, CLIP-like multimodal models, and point-cloud transformers as part of a shared but internally diverse family (Mia et al., 2023). Subsequent work then specializes individual components—patch embedding, attention computation, positional encoding, skip structure, upsampling, and adaptation modules—to fit particular data or deployment constraints.

2. Training ViTs with limited data

Small-data behavior is one of the most intensively studied fault lines in the ViT literature. In “Bootstrapping ViTs,” the stated problem is that ViTs overfit and underperform on small datasets without large-scale pretraining because they lack hard-coded inductive biases such as locality and weight sharing. The proposed remedy is an agent CNN matched to the ViT’s block structure, with generalized convolution replacing dense attention and with shared weights where possible; the ViT is then trained jointly with the agent through mutual learning and intermediate feature supervision (Zhang et al., 2021). On CIFAR-10 and CIFAR-100, this changes ViT-B from 79.24% and 53.07% to 95.00% and 77.83% without pretraining, while the paper also reports 36.01% on 5% of ImageNet for the joint method versus 16.6% for vanilla ViT-B (Zhang et al., 2021).

A separate line of work replaces sequential self-supervised pretraining and supervised fine-tuning with joint optimization. The SSAT formulation combines the primary supervised objective and a self-supervised auxiliary task,

$$L = \lambda \cdot L_{\text{cls}} + (1 - \lambda) \cdot L_{\text{SSAT}},$$

and reports that reconstruction-based masked autoencoding is the most effective auxiliary signal among the tested SSL families (Das et al., 2023). On ImageNet-1K, ViT-T improves from 65.0% to 72.7% and ViT-S from 74.2% to 76.4%; on CIFAR-10, ViT-T improves from 79.5 to 91.7, and on CIFAR-100 from 55.1 to 69.6 (Das et al., 2023). The paper further states that this joint regime reduces carbon footprint relative to sequential SSL plus fine-tuning, with ViT-T on ImageNet-1K requiring 78 GPU hours and 7.55kg CO$_2$e versus 104 GPU hours and 10.35kg CO$_2$e (Das et al., 2023).

Label-aware contrastive fine-tuning addresses a related limitation from a different angle. LaCViT keeps the ViT architecture unchanged but inserts a first-stage contrastive objective in which positives are all same-class embeddings in the minibatch and negatives are all embeddings from different classes, followed by standard task-head fine-tuning (Long et al., 2023). The paper reports statistically significant Top-1 gains across three evaluated ViTs and eight datasets, with the largest cited improvement being MAE on CUB-200-2011 from 78.46% to 89.24%, a 10.78% absolute increase; ViT-B on CIFAR-100 improves from 87.13% to 91.01% (Long et al., 2023). In the language of that work, the improvement is attributed to tighter intra-class clustering and higher inter-class separation.

The CIFAR-10 tiny-ViT study makes the small-data recipe more granular. Data augmentation is described as fundamental: removing AutoAugment drops accuracy from 93.65% to 92.14%, removing CutMix drops it to 92.77%, and removing MixUp slightly improves it to 93.68%, which the authors interpret as a sign of possible over-regularization (Wu, 7 Jan 2025). The same paper reports that compressing the patch-token dimension from 192 to 96 reduces accuracy from 93.65% to 90.41%, but adding a second CLS token at dimension 96 raises accuracy to 93.95%, exceeding the baseline (Wu, 7 Jan 2025). This suggests that small-data ViT design is governed not by a single intervention but by a coupled set of augmentation, positional, token-capacity, and optimization decisions.

3. Efficiency, compression, and flexible compute allocation

The computational overhead of ViTs motivates a large compression and runtime-adaptation literature. Unified Visual Transformer Compression formulates pruning, layer skipping, and knowledge distillation as a joint budget-constrained optimization problem, solved with a primal-dual algorithm (Yu et al., 2022). The paper reports that DeiT-Tiny with distillation can be reduced from 1.3G FLOPs to 0.66G FLOPs, or 50.6% of the original computation, with Top-1 accuracy changing from 74.4% to 74.1%; for DeiT-Small, 4.6G FLOPs can be reduced to 2.32G with accuracy changing from 79.8% to 78.82% (Yu et al., 2022). The same work argues that skip manipulation alone is unstable and that joint end-to-end optimization is essential.

IdleViT addresses efficiency at the token level. Rather than permanently pruning image tokens, it selects a subset of tokens to be active in a given layer and lets the remaining tokens bypass the layer as idle tokens, with the possibility of re-selection in later layers (Xu et al., 2023). The stated motivation is to avoid irreversible information loss from early pruning. The paper reports that complexity can be reduced by up to 33% with no more than 0.2% accuracy decrease on ImageNet after only 30 epochs of fine-tuning, and that at keep ratio 0.5 IdleViT on DeiT-S achieves 79.0% Top-1, outperforming EViT by 0.5% while also running faster at inference (Xu et al., 2023). A further result is that up to roughly 36% of the tokens used in the final layer can be re-selected tokens, which is presented as evidence of self-correction.

Compute flexibility can also be obtained without committing to a single fixed backbone. SN-Netv2 stitches pretrained ViTs of different sizes into a single flexible model, extending the stitching space with two-way stitching, balancing optimization with resource-constrained sampling, and stabilizing downstream adaptation by learning stitching layers as low-rank updates (Pan et al., 2023). The paper states that the enlarged stitching space is 10× larger, covers hundreds of intermediate trade-offs, and can replace multiple separately adapted anchor models. One explicit example is ADE20K training for DeiT3-S/L, where SN-Netv2 requires 120 A100 GPU hours versus 264 for separate anchor trainings (Pan et al., 2023).

Architectural inductive bias can itself be used as a route to efficiency. Octic-equivariant ViTs incorporate equivariance to reflections and 90-degree rotations and are evaluated in both DeiT-III and DINOv2 settings (Nordström et al., 21 May 2025). The paper reports approximately 40% reduction in FLOPs for ViT-H while improving performance; for ViT-H/14 under DeiT-III, FLOPs drop from 167.8G to 102.3G while Top-1 accuracy increases from 84.6% to 85.0% in the octic hybrid configuration (Nordström et al., 21 May 2025). The same study states that the modified linear layers enjoy substantial structured savings because equivariance constrains cross-irrep mixing.

4. Adaptation, forgetting, and post hoc modification

Fine-tuning a pretrained ViT can be destructive. In the catastrophic-forgetting study on self-supervised ViTs, a DINO ViT-Base/16 pretrained on ImageNet-1k is reported to lose over 70% accuracy on ImageNet-1k after just 10 iterations of fine-tuning on CIFAR-100 (Bafghi et al., 2024). Two parameter-efficient strategies are then evaluated: Block Expansion, which inserts identity-initialized trainable blocks while freezing the original backbone, and LoRA, which adds low-rank adapters to attention projections using

$W' = W + AB.$

On CIFAR-100, full fine-tuning gives 88.13% on the new task but only 25.24% on ImageNet-1k retention, whereas Block Expansion with $p=3$ blocks gives 88.58% on CIFAR-100 and 74.61% on ImageNet-1k, and LoRA with $r=8$ gives 88.27% and 65.99% respectively (Bafghi et al., 2024). Within that comparison, Block Expansion is presented as the most favorable stability–plasticity compromise.

Post hoc architectural modification is also possible without full retraining. “Vision Transformers with Self-Distilled Registers” addresses artifact tokens—dense patch representations that are not semantically coherent with their local image regions and that degrade fine-grained tasks such as segmentation and depth prediction (Chen et al., 27 May 2025). The method, PH-Reg, keeps a frozen teacher ViT unchanged, augments a student copy with randomly initialized register tokens, denoises teacher features via test-time augmentation, and optimizes only a small subset of student weights through self-distillation (Chen et al., 27 May 2025). On open-vocabulary segmentation, the paper reports mIoU values of 63.0 on VOC21, 37.2 on City, and 19.2 on ADE, with an average of 41.9, improving over the listed SCLIP average of 40.1 (Chen et al., 27 May 2025). The method is explicitly designed to avoid the need for labeled data or full retraining of large pretrained ViTs.

A plausible implication of these two lines of work is that pretrained ViTs are increasingly treated as mutable but protected backbones: either one confines adaptation to auxiliary modules to preserve pretrained knowledge, or one distills corrective structure into the model while leaving the majority of parameters fixed.

5. Dense prediction, variable resolution, and medical imaging

Dense prediction exposes a structural bottleneck in standard ViTs: coarse patch-token grids. ViT-Up frames feature upsampling as implicit prediction at arbitrary continuous image coordinates, using layer-wise query refinement from intermediate ViT hidden states rather than shallow image-guided upsampling (Wandel et al., 12 Jun 2026). The paper argues that image-guided approaches can introduce feature leakage, fragmentation, and blur, whereas backbone-aligned query construction preserves the ViT feature space. Quantitatively, on DINOv3-S+ ViT-Up improves over prior methods by up to +2.07 mIoU on Cityscapes and +4.17 [email protected] on SPair-71k; with DINOv3-B, the gains rise to +3.36 mIoU and +8.09 [email protected] (Wandel et al., 12 Jun 2026).

VariViT addresses a different spatial limitation: the fixed input-size assumption. It keeps patch size constant while allowing the number of patches to vary with image size, uses a center-and-select positional embedding resizing scheme instead of interpolation, and introduces two batching strategies—Custom Batch Sampler and Gradient Accumulation—to exploit variable input dimensions efficiently (Varma et al., 16 Feb 2026). On two 3D brain MRI tasks, VariViT reports F1-scores of 75.5% and 76.3%, while its batching strategy reduces computation time by up to 30% (Varma et al., 16 Feb 2026). The paper further reports that VariViT-GA reaches AUC 0.942 and F1 0.755 for IDH mutation status classification, versus AUC 0.927 and F1 0.744 for vanilla ViT (Varma et al., 16 Feb 2026).

In medical segmentation, calibrated self-supervised ViTs have been evaluated on heterogeneous 3D CT data for intracranial arterial calcification. The study pretrains 3D ViTs with masked autoencoders on 4933 unlabeled regions of interest, masking 90% of patches for 800 epochs, and then fine-tunes on 158 manually segmented regions (Jin et al., 2 Jul 2025). The reported result is that the calibrated MAE-pretrained ViT exceeds a supervised nnU-Net baseline by 3.2 Dice points, with Dice 65.1 versus 62.1 and volume difference 100 mm$x_\text{token} = x_e + x_p,$0 versus 188 mm$x_\text{token} = x_e + x_p,$1 (Jin et al., 2 Jul 2025). The same study states that small patch sizes are crucial, that interpolation upsampling with regular convolutions is preferable to transposed convolutions for ViT-based models, and that calibrated ViTiac-S improves clinical risk-group classification from 105/209 cases to 153/209, a 46% increase (Jin et al., 2 Jul 2025).

Taken together, these results indicate that dense ViT deployment depends on at least three interacting design axes: the patch-grid resolution exposed by the backbone, the faithfulness of any feature upsampler, and the ability to accommodate variable image size or volume geometry without destructive resizing.

6. ViTs beyond still images: video and action analysis

Video extensions of ViTs preserve the token-sequence viewpoint while modifying the tokenization strategy. TubeViT turns a ViT encoder into a joint image-and-video model by sparsely sampling spatiotemporal tubes of different shapes, strides, and offsets rather than densely tokenizing every frame (Piergiovanni et al., 2022). For a $x_\text{token} = x_e + x_p,$2 video input, the paper reports a total of only about 559 tokens using its sparse tube configurations, and it emphasizes that the same encoder can process both image patches and video tubes (Piergiovanni et al., 2022). On benchmarks, TubeViT-L reaches 90.2% Top-1 on Kinetics-400 and 83.8% on Kinetics-700, while TubeViT-H reaches 90.9% on Kinetics-400 and 91.8% on Kinetics-600 (Piergiovanni et al., 2022). The same paper further reports that partial fine-tuning of a large ViT-H requires 43% fewer updates than full fine-tuning while approaching the same performance (Piergiovanni et al., 2022).

A specialized application is risky-tackle detection in American football practice videos. The ViViT-B $x_\text{token} = x_e + x_p,$3 architecture is used on 733 temporally localized clips labeled with a standardized strike-zone rubric, with imbalance-aware training based on focal loss and systematic augmentation design via Taguchi’s L18 orthogonal array (Zaidi et al., 1 Apr 2026). The paper reports risky recall 0.67 and Risky F1 0.59 under 5-fold stratified cross-validation, improving over a prior C3D baseline on a smaller subset that achieved risky recall 0.58 and Risky F1 0.56 (Zaidi et al., 1 Apr 2026). A notable procedural claim is that augmentation is applied only after stratified fold splitting and only to training data, mainly to risky examples, while validation remains unaugmented (Zaidi et al., 1 Apr 2026).

These studies show two distinct trajectories for video ViTs. One trajectory seeks a unified image–video backbone through sparse spatiotemporal tokenization and partial adaptation of large image-pretrained models (Piergiovanni et al., 2022). The other treats a ViT-based video model as a task-specific recognizer whose effectiveness depends on temporal localization, class-imbalance handling, and operating-threshold selection for the deployment objective (Zaidi et al., 1 Apr 2026).

7. Robustness, perception, and adversarial behavior

The robustness profile of ViTs is heterogeneous rather than uniformly favorable. In the corruption-benchmarking study, overlapping patch embedding and convolutional FFNs are identified as simple architectural modifications that consistently improve robustness on ImageNet-C and ImageNet-3DCC (Tian et al., 2022). For DeiT-S, mean Corruption Error drops from 54.60 to 52.40 with overlapping patch embedding and to 52.41 with depth-wise convolution in the FFN; combining overlapping patch embedding, convolutional FFN, and conditional adversarial noise training yields mCE 45.33 (Tian et al., 2022). At larger scale, DeiT-B with conditional ANT reaches ImageNet-C mCE 40.68, ImageNet-3DCC accuracy 65.31%, and clean ImageNet accuracy 82.35% (Tian et al., 2022). The same paper reports that adversarial noise training is powerful for ViTs, whereas Fourier-domain augmentation is inferior.

Robustness to visual corruption does not imply human-like low-level visual judgment. In a controlled study of graphical perception tasks based on Cleveland and McGill, ViTs are benchmarked against CNNs and humans on encodings such as position, length, angle, area, curvature, shading, and point-cloud estimation (Poonam et al., 20 Feb 2026). Swin is reported as the best ViT, but it still lags in several visualization-relevant tasks: on point cloud, the best ViT obtains MLAE 6.37 versus 3.40 for VGG19 and 4.95 for humans; on bars and framed rectangles, the best ViT obtains 4.75 versus 1.93 for both humans and the best CNN (Poonam et al., 20 Feb 2026). The paper concludes that alignment with human-like graphical perception is limited.

Adversarial transferability reveals a different aspect of ViT behavior. The redundancy-focused attack study argues that adversarial examples crafted on ViTs transfer more effectively than those crafted on CNNs because ViTs contain exploitable data-level and model-level redundancy (Liu et al., 15 Apr 2025). The paper proposes attention sparsity manipulation, attention head permutation, clean token regularization, ghost MoE diversification, and test-time adversarial training, and reports on ImageNet-1K that its full method reaches average fooling rate 86.9 across CNN and transformer targets, compared with 77.8 for PGN and 54.1 for MI-FGSM (Liu et al., 15 Apr 2025). On ViT-B/16 specifically, the reported fooling rate is 99.7 (Liu et al., 15 Apr 2025).

A neutral synthesis of these studies is that “robustness” is not a unitary property for ViTs. Architectural tweaks and conditional augmentation can improve resistance to common corruptions (Tian et al., 2022), yet human-aligned graphical perception remains incomplete (Poonam et al., 20 Feb 2026), and the same structural properties that enable efficiency or flexibility may also create favorable conditions for highly transferable adversarial attacks (Liu et al., 15 Apr 2025).