Enhanced Vision Transformer (ViT++)

• Enhanced Vision Transformer (ViT++) is a family of derivative models that address limitations of vanilla ViT, such as quadratic attention cost and weak locality bias.
• Hierarchical designs and convolutional modifications are key strategies that inject CNN-like multi-scale processing and local continuity into transformer architectures.
• Training-time and internal block enhancements, including refined Q/K/V embeddings and specialized feed-forward modules, boost performance on specific vision tasks.

Enhanced Vision Transformer (“ViT++” as an Editor’s term) denotes an informal family of Vision Transformer extensions rather than a single canonical architecture. The surveys in this corpus explicitly note that there is no model literally defined as “ViT++”; instead, the literature presents a set of derivative designs that modify patch embedding, positional encoding, attention, feed-forward layers, stage hierarchy, training objectives, or deployment mechanisms to address limitations of vanilla ViT such as quadratic attention cost, weak locality bias, fixed-resolution assumptions, and difficulty on dense prediction or low-data regimes (Islam, 2022, Fu, 2022).

1. Baseline formulation and the rationale for enhancement

The canonical ViT pipeline described in the surveys splits an image into non-overlapping or overlap patches, linearly projects flattened patches into patch embeddings, adds positional embedding and a class token, feeds the token sequence into transformer encoder layers, and uses the class token output and an MLP head for final prediction (Islam, 2022). In the standard formulation, self-attention is

$$\mathrm{Attention}(\mathbf Q,\mathbf K,\mathbf V)=\mathrm{softmax}\!\left(\frac{\mathbf Q\mathbf K^\top}{\sqrt{d_k}}\right)\mathbf V,$$

with multi-head attention and MLP sublayers stacked in repeated encoder blocks (Islam, 2022).

Across the reviewed papers, the baseline limitations are consistent. Vision Xformers emphasizes that attention scales quadratically with input-sequence length and that transformers lack an inductive bias that is appropriate for images (Jeevan et al., 2021). ViT-ResNAS argues that standard ViT and DeiT inherit a flat, constant-resolution design in which sequence length stays constant through depth, unlike CNN pyramids that reduce spatial resolution while increasing channel width (Liao et al., 2021). The short review of ViT derivatives repeatedly returns to the same pressure points: high computational cost, weak local inductive bias, and limited suitability of plain global-attention backbones for dense prediction (Fu, 2022).

This suggests that “enhancement” in the ViT literature is best understood as a family of targeted responses to those structural weaknesses rather than as a single replacement backbone.

2. Hierarchy, locality, and efficient token processing

A dominant enhancement direction is to make ViT more hierarchical and locality-aware. The review of derivatives identifies Pyramid Vision Transformer (PVT), PVT-v2, Swin Transformer, MViT, and SegFormer as key examples. PVT introduces a pyramidal backbone and Spatial-Reduced Attention; PVT-v2 adds overlapping patch embedding, convolutional feedforward networks, and linear-complexity self-attention layers; Swin Transformer uses shifted windows and a hierarchical structure; MViT builds a multi-scale feature pyramid; and SegFormer uses a hierarchical Transformer encoder with a simple MLP decoder and no positional encoding (Fu, 2022).

A related line reworks stage transitions explicitly. ViT-ResNAS proposes residual spatial reduction, decreasing sequence lengths for deeper layers and using a multi-stage architecture with skip connections during reduction; its manually designed ViT-Res-Tiny uses three stages with sequence progression $$16\times16+1 \rightarrow 8\times8+1 \rightarrow 4\times4+1$$, while the searched ViT-ResNAS family improves the accuracy-MACs and accuracy-throughput trade-offs over DeiT and other strong baselines on ImageNet (Liao et al., 2021). HIRI-ViT pushes the same idea toward high-resolution input scaling by upgrading prevalent four-stage ViT to five-stage ViT tailored for high-resolution inputs, using an HR-stem, HR blocks in the first two stages, CFFN in stage 3, Transformer blocks in stages 4 and 5, and IRDS downsampling modules (Yao et al., 2024).

Another branch injects locality through convolution without abandoning token attention. Vision Xformers replaces the initial linear embedding layer with three convolutional layers and also studies efficient attention replacements such as Performer, Linformer, and Nyströmformer, alongside RoPE, to reduce memory and improve classification accuracy in long-sequence settings (Jeevan et al., 2021). Vision Conformer (ViC) goes further inside the encoder block: it replaces the MLP role with a CNN, but first reconstructs image-like features from token embeddings through reverse embedding, then applies convolution, then patchifies and re-embeds before continuing Transformer processing (Iwana et al., 2023).

Enhancement family	Representative models	Main modification
Hierarchical efficient backbones	PVT, PVT-v2, Swin, MViT, SegFormer	Pyramids, windowed or reduced attention, multi-scale features
Residual stage redesign	ViT-ResNAS, HIRI-ViT	Residual spatial reduction, five-stage high-resolution layouts
Convolutional locality injection	Vision Xformers, ViC	Convolutional embedding or convolution inside transformer layers

A plausible implication is that many “ViT++” systems recover, in different forms, the CNN heuristics of multi-scale processing and local continuity while retaining Transformer-style token interaction.

3. Internal attention and feed-forward reparameterization

A second enhancement axis revisits the internals of the transformer block itself. One example is the query-key-value embedding stage. “Rethinking Query, Key, and Value Embedding in Vision Transformer under Tiny Model Constraints” proposes Separate Non-linear Embedding (SNE), Partially-Shared Non-linear Embedding (P-SNE), and Fully-Shared Non-linear Embedding (F-SNE), replacing the usual linear $Q/K/V$ projections with richer non-linear structures and, in the fully shared case, trainable role codes. The proposal is evaluated mainly on XCiT-N12 and XCiT-T12, and the paper argues that richer, better-structured $Q/K/V$ embedding can improve accuracy without materially increasing parameter count (Ahn et al., 2021).

Other work constrains or propagates attention more explicitly. O-ViT replaces standard $Q/K/V$ projections with orthogonally parameterized ones so that the self-attention projections lie on the orthogonal manifold. Its central mapping is

$h(X)=2(E+X)^{-1}-E,$

with skew-symmetric parameterization used to guarantee orthogonality while keeping Euclidean-space optimization (Fei et al., 2022). ReViT addresses feature collapse in deeper layers by propagating pre-softmax attention scores across adjacent layers. For $l>0$, it defines

$$S_l=\alpha\left(\frac{Q_lK_l^T}{\sqrt{d_l}}\right)+(1-\alpha)S_{l-1},$$

so residual attention carries patch-to-patch relational structure rather than only token features (Diko et al., 2024).

The feed-forward path is another major target. Hyb-KAN ViT replaces MLP layers with Efficient-KAN and Wavelet-KAN modules, using Wav-KAN in the encoder and Eff-KAN in the head in its best-performing “Hybrid-1” configuration. The paper frames this as upgrading the weakest part of vanilla ViT—the MLP/feed-forward subnetwork—through spline-parameterized nonlinear mappings and wavelet-based multiresolution processing (Dey et al., 7 May 2025). Proximal Vision Transformer, by contrast, keeps the standard ViT frontend and adds a batch-level proximal/self-representation module over the final class-token features, refining the classifier input from $\mathbf Z$ to $\hat{\mathbf Z}=\mathbf Z\mathbf W$ through unfolded proximal iterations (Yun et al., 23 Aug 2025).

Taken together, these methods indicate that “enhancement” often targets not only attention complexity or hierarchy, but also the semantics of $$16\times16+1 \rightarrow 8\times8+1 \rightarrow 4\times4+1$$0 construction, the geometry of attention projections, the persistence of attention maps through depth, and the expressive power of the feed-forward sublayer.

4. Training-time and objective-level enhancements

Not all enhanced ViTs are architectural. Several important variants keep the deployed backbone largely unchanged and instead alter supervision, optimization, or pretext structure.

The review’s “Scaling Vision Transformer” section describes a slightly modified ViT with 2 billion parameters, removal of the class token in favor of global average pooling or multi-head attention pooling, decoupled weight decay for the head and the body, warm-up and cool-down learning-rate schedules, and Adafactor in place of Adam. In that setting, the scaled model achieves 90.45% top-1 accuracy on ImageNet and 84.86% top-1 accuracy on ImageNet in the 10-examples-per-class few-shot setting (Fu, 2022).

Jigsaw-ViT adds a self-supervised auxiliary loss tailored to ViT’s patch-token representation. The training objective is

$$16\times16+1 \rightarrow 8\times8+1 \rightarrow 4\times4+1$$1

where the jigsaw branch predicts absolute patch positions from the final patch tokens, discards positional embeddings in that branch, and uses random patch masking. The paper presents this as a way to improve both generalization and robustness without redesigning the transformer encoder itself (Chen et al., 2022).

HAT is another training-only enhancement. It begins from the hypothesis that ViT models are less effective in capturing the high-frequency components of images than CNN models, then uses adversarial training to augment high-frequency components. Its final objective keeps the clean loss and adds an adversarial term with $$16\times16+1 \rightarrow 8\times8+1 \rightarrow 4\times4+1$$2-bounded perturbations, plus a symmetric KL regularizer between clean and perturbed predictions (Bai et al., 2022).

This family of methods shows that a substantial part of the “ViT++” literature operates at the level of representation shaping and training dynamics rather than only at the level of encoder topology.

5. Task-specific and systems-level interpretations of enhancement

Several papers use “enhanced ViT” in a deliberately task-specific sense. Case-Enhanced Vision Transformer (CEViT) is explicit that it is not a general-purpose replacement for standard Vision Transformers. It concatenates a query image and a reference image along the channel dimension to shape $$16\times16+1 \rightarrow 8\times8+1 \rightarrow 4\times4+1$$3, predicts a scalar same-class score in $$16\times16+1 \rightarrow 8\times8+1 \rightarrow 4\times4+1$$4, and then serves as a learned similarity metric inside $$16\times16+1 \rightarrow 8\times8+1 \rightarrow 4\times4+1$$5-nearest neighbors classification. Its explainability claim comes from CLS-to-patch attention masks that are conditioned on a specific prior case, yielding case-conditioned explanations rather than fixed saliency maps (Zhao et al., 2024).

In low-level vision, “Image Reconstruction using Enhanced Vision Transformer” defines an enhanced ViT as a reconstruction-oriented framework that augments a vanilla ViT with Shifted Patch Tokenization (SPT), Rotary Position Embeddings (RoPE), Locality Self-Attention (LSA), and an adversarial loss with a ViT-based discriminator for denoising and inpainting (Verma et al., 2023). In robotics, VAT extends a pretrained ViT by threading action tokens through the full representation trajectory of the vision backbone so that action generation is conditioned on features from all transformer layers rather than only the final layer (Li et al., 3 Dec 2025).

At the systems end, Opto-ViT is “ViT++” only in a deployment-centric sense. It keeps a mostly standard ViT backbone, adds a lightweight Mask Generation Network (MGNet) for region-of-interest pruning, uses 8-bit quantization-aware training, and maps heavy matrix multiplications to a hybrid silicon-photonic accelerator near the sensor (Morsali et al., 9 Jul 2025). The short review also makes clear that cross-applications such as TimeSformer for video and UNETR for 3D medical image segmentation are part of the derivative landscape, even though their modifications are domain-specific rather than universal image-classification upgrades (Fu, 2022).

This suggests that the phrase “enhanced Vision Transformer” is frequently local to a problem setting: explainable retrieval, reconstruction, robotic policy learning, segmentation, or hardware-aware deployment.

6. Empirical profile, common misconceptions, and limitations

The empirical evidence in this corpus is strong but heterogeneous. In tiny-model classification, richer $$16\times16+1 \rightarrow 8\times8+1 \rightarrow 4\times4+1$$6 embedding yields measurable gains: the best ImageNet-1k result in the QKV paper is 71.4% with 3.1M parameters versus 69.9% for XCiT-N12, and the best transfer-learning average is 93.3% with 2.9M parameters versus 92.2% for baseline (Ahn et al., 2021). In high-resolution hierarchical design, HIRI-ViT-S reaches 84.3% top-1 on ImageNet with 448×448 inputs at approximately 5.0 GFLOPs, improving 83.4% of iFormer-S with 224×224 inputs by 0.9% under comparable computational cost (Yao et al., 2024). In training-time enhancement, HAT improves ViT-B by +1.2%, Swin-B by +0.5%, and raises VOLO-D5 to 87.3% using only ImageNet-1K data (Bai et al., 2022). In attention-residual design, ReViT-B reports 82.4 top-1 on ImageNet1K and is described as +4.6% over its original counterpart while keeping 86M parameters and 17.5G FLOPs (Diko et al., 2024).

At the same time, the literature repeatedly warns against treating any one of these systems as a universal “better ViT.” CEViT is preliminary research on MNIST, with 99.0% classification accuracy for CEViT + k-NN versus 99.1% for ViT and 97.1% for Euclidean-distance k-NN, and the authors explicitly state that it should not be read as a general improvement over ViT for classification, detection, or segmentation (Zhao et al., 2024). Opto-ViT reports 100.4 KFPS/W with up to 84% energy savings and less than 1.6% accuracy loss, but its contribution is largely hardware-software co-design rather than a new software-only transformer family (Morsali et al., 9 Jul 2025).

A common misconception is therefore that “ViT++” names a single architecture with consistent superiority across all vision tasks. The surveys and case studies do not support that reading. They support a narrower and more technical interpretation: “enhanced ViT” denotes a broad research program in which ViT is modified along several largely orthogonal axes—hierarchy, locality, efficient attention, internal block geometry, auxiliary objectives, task reformulation, and deployment co-design—each with its own evidence, constraints, and scope of validity (Islam, 2022).