Patch-based Vision Transformer (ViT)

• Patch-based Vision Transformer (ViT) is a model that represents images as non-overlapping patch sequences, enabling global self-attention for visual tasks.
• It leverages self-attention to capture local and global dependencies, thereby enhancing robustness and accuracy in various vision applications.
• Advanced techniques like patch selection, slimming, and hierarchical merging reduce computational costs while maintaining high performance.

A Patch-based Vision Transformer (ViT) is a transformer architecture for visual data in which images are represented as sequences of non-overlapping patches, each patch serving as an independent token to the transformer model. This patchification paradigm enables self-attention networks, originally architected for NLP, to directly address images as sequences, thereby leveraging cross-patch dependency modeling, sparse or dense interaction strategies, and diverse token manipulation operations for a variety of vision workloads. Patch-based ViTs exhibit unique algorithmic features and challenges in terms of computation, representation, robustness, efficiency, and explainability, which has driven a rich recent research literature.

1. Patch Extraction and Token Embedding

The canonical ViT pipeline begins by dividing an image $x \in \mathbb{R}^{H \times W \times C}$ into a regular grid of $N=(H/P)\cdot (W/P)$ non-overlapping patches, each of size $P \times P \times C$ ($P$ e.g. 16, 32). Each patch $x_i$ is flattened ($\in \mathbb{R}^{P^2 C}$) and mapped linearly to an embedding $z_i^0=W_e x_i + b_e$, $W_e \in \mathbb{R}^{D \times P^2 C}$. A learnable class token $z_\text{cls}^0$ is prepended, and a 1-D or 2-D positional encoding $E_{pos}$ added: $N=(H/P)\cdot (W/P)$0. This sequence serves as input to $N=(H/P)\cdot (W/P)$1 layers of standard transformer encoder blocks implementing multi-head self-attention (MSA), MLPs, and normalization. Patch-based encoding is the foundation for ViT’s architecture and its variants, enabling a range of token-level operations such as selection (Kinfu et al., 2023), merging (Yu et al., 2024), and invariance learning (Yun et al., 2022).

2. Patch Interactions, Self-Attention, and Representation Learning

The distinguishing feature of ViT is its global, non-local self-attention across all patches. For each transformer block, token projections are computed:

$N=(H/P)\cdot (W/P)$2

and the attention map is

$N=(H/P)\cdot (W/P)$3

resulting in outputs $N=(H/P)\cdot (W/P)$4. Patch-based MSA enables ViT to contextualize local patch features by aggregating over all patches, resulting in strong non-local representations suited to global pattern recognition. However, the quadratic cost $N=(H/P)\cdot (W/P)$5 in $N=(H/P)\cdot (W/P)$6 patches motivates approximate or hierarchical methods.

Several recent works enrich the patch-based paradigm:

• Patch-to-cluster attention (PaCa-ViT): Rather than all-to-all self-attention, PaCa clusters $N=(H/P)\cdot (W/P)$7 patches into $N=(H/P)\cdot (W/P)$8 semantic groups, then attends from patches to clusters, reducing complexity and increasing interpretability (Grainger et al., 2022).
• Patch-level invariance: SelfPatch enforces that each patch’s embedding is invariant to semantic neighbors, using a local patch-level loss atop transformer outputs, improving local representation structure for dense prediction (Yun et al., 2022).
• Multi-scale/Hierarchical merging: Stepwise Patch Merging (SPM) introduces multi-scale local aggregations and guided local enhancement before spatial downsampling, balancing local and global cue integration (Yu et al., 2024).

3. Efficiency: Patch Selection, Slimming, and Compression

The quadratic scaling of self-attention with the number of patches motivates methods for token sparsification:

• Patch selection: For dense-prediction tasks, only contextually relevant patches (e.g., near predicted joints in pose estimation) are processed beyond early layers, yielding up to 44% FLOP reduction with minimal (~1%) accuracy loss (Kinfu et al., 2023). Approaches include pose-guided neighbor selection, joint-token–based dynamic pruning, and skeleton-line patch filtering.
• Patch slimming: A top-down, layer-wise mask selection eliminates tokens whose contributions to the final class token are estimated negligible based on attention cascades, achieving up to 50% FLOP cuts for <0.5% top-1 accuracy loss (Tang et al., 2021).
• Patch summarization: BUS (Bottom-up Patch Summarization) integrates a text-semantics-aware patch selector and a patch abstraction decoder to reduce the visual token sequence length during vision-language pretraining by >80% with negligible loss (Jiang et al., 2023).
• Compression-based embedding: CI2P-ViT replaces the standard patch projection with a frozen, pretrained CNN compressor. The output is reshaped and projected into a reduced set of patch tokens ($N=(H/P)\cdot (W/P)$9 original) for ViT, cutting FLOPs by 63% while boosting accuracy ($P \times P \times C$0 on Animals-10) (Zhao et al., 14 Feb 2025).

4. Robustness: Occlusion, Adversariality, and Equivariance

Patch-based ViTs present unique robustness properties:

• Patch selectivity: ViTs' non-local attention naturally enables disregarding irrelevant (out-of-context) patches, conferring strong occlusion robustness compared to CNNs. PatchMixing explicitly forces CNNs to acquire patch selectivity similar to ViTs (Lee et al., 2023).
• Patch perturbations: ViTs are more robust than CNNs to natural patch occlusions but more vulnerable to adversarial patches, which can hijack the attention map (Gu et al., 2021). Simple post-hoc temperature scaling of the attention softmax restores robustness with minimal clean accuracy degradation.
• Negative augmentation: Training ViTs to assign uniform/conflicting labels to semantically destructive patch transforms (shuffle, rotate, infill) increases robustness to distribution shifts and encourages less reliance on non-robust local cues (Qin et al., 2021).
• Certifiable defense: For physical patch attacks, ViT can be combined with derandomized smoothing and progressive modeling over band-like regions, enabling certified accuracy under adversarial patches up to ImageNet scale (e.g., $P \times P \times C$1 certified on 2% patch attacks, compared to previous $P \times P \times C$2) (Chen et al., 2022).
• Rotational equivariance: Equi-ViT integrates group-equivariant convolutions into the patch embedding step to enforce continuous rotation equivariance for all tokens, yielding both higher mean accuracy and vastly reduced rotation variance in histopathology benchmarks (Chen et al., 14 Jan 2026).

5. Advanced Techniques: Patch Processing Variants

Contemporary research extends the basic patch paradigm:

• Multi-scale and frequency pyramids: RetinaViT injects patches from multiple downsampled versions of the input image—concatenated as additional tokens and provided with scale-sensitive positional embeddings—improving accuracy by $P \times P \times C$3 (ImageNet-1K) and robustness on distribution shifts, for ~5% extra params (Shu et al., 2024).
• Resolution/aspect-ratio flexibility: NaViT (Native Resolution ViT) supports arbitrary aspect ratios and resolutions using packed token sequences and factorized positional embeddings, attaining state-of-the-art cost-accuracy/fairness trade-offs, and continuous resource scaling at inference (Dehghani et al., 2023).
• Semantic-aware grouping: PaCa-ViT’s patch-to-cluster formulation simultaneously creates a learnable, semantic tokenizer and removes quadratic cost barriers, achieving or surpassing state-of-the-art in classification, detection, and segmentation, with explicit heatmap-based interpretability (Grainger et al., 2022).
• Task-tailored tokenization: PaW-ViT replaces default grids with anatomy-aware, warped patch layouts for ear biometrics, increasing robustness to variation by aligning grid sectors to semantic boundaries without modifying the transformer itself (Arun et al., 27 Jan 2026).
• Patch-based self-supervision: Jigsaw-ViT formulates a jigsaw auxiliary loss (re-predicting absolute positions under random masking, without explicit positional encoding), yielding simultaneous gains in generalization (e.g. +0.7% on ImageNet) and robustness (noisy labels, adversarial attacks) (Chen et al., 2022).

6. Explainability and Visualization of Patch Interactions

The structure and dynamics of patch interactions in ViTs remain an active area of research. Approaches include:

• Attention rollout and patch saliency: Rollout of multi-layer attentions quantifies the effective receptive field of each patch and its contribution to the class token (Gu et al., 2021).
• Patch impact estimation: Patch Slimming and related methods compute per-patch importance scores layer-wise through attention path tracing (Tang et al., 2021).
• Cluster assignment heatmaps: PaCa-ViT’s attention structure allows mapping cluster assignments to spatial heatmaps, directly localizing salient regions (Grainger et al., 2022).
• Multi-head selection for fine-grained cues: ViT-FOD fuses class tokens from multiple depths (CTI), employs critical region filtering to discard uninformative patches, and recombines informative patches from multiple images (APC), systematically dissecting where and how ViT attends for subtle discrimination (Zhang et al., 2022).
• Joint visualization with text: BUS integrates attention-based and MLP-based text-conditioned patch scoring, enabling identification and pruning of text-irrelevant patches in multi-modal ViT pipelines (Jiang et al., 2023).

7. Applications and Benchmark Performance

Patch-based ViTs and their derivatives have dominated a broad spectrum of vision and vision-language tasks:

• Dense prediction: SelfPatch, SPM, and patch merging/selection approaches deliver consistent mIoU and AP gains across semantic segmentation, object detection, and video segmentation benchmarks (Yun et al., 2022, Yu et al., 2024, Kinfu et al., 2023).
• Fine-grained classification: ViT-FOD achieves top-1 accuracy improvements up to +2.7% on challenging bird and dog benchmarks by precise patch reweighting (Zhang et al., 2022).
• Vision-language pre-training: BUS allows high-resolution, efficient pre-training for VQA, captioning, and retrieval, with up to 50% throughput gains at state-of-the-art accuracy (Jiang et al., 2023).
• Robustness and certification: Novel patch-based pretext/auxiliary tasks and sparsity-aware token strategies have realized SOTA OOD, occlusion, and certifiable adversarial robustness metrics on ImageNet and large-scale biomedical datasets (Qin et al., 2021, Chen et al., 2022, Chen et al., 14 Jan 2026).
• Specialized domains: Anatomical warping (PaW-ViT) and rotational equivariance (Equi-ViT) demonstrate that patch tokenization strategies can be adapted to prior-informed, domain-specific divisors for enhanced robustness and expressivity (Arun et al., 27 Jan 2026, Chen et al., 14 Jan 2026).

---

Patch-based Vision Transformers constitute a modular, extensible, and semantically interpretable family of models. Their flexibility in patch representation, interaction, and manipulation has enabled dramatic advances across accuracy, robustness, efficiency, and explainability axes, with continued innovation in token compression/selection, cluster- or semantic-aware attention, and biologically inspired patch processing strategies.