---
title: Native Vision Transformer (ViT) Overview
url: https://www.emergentmind.com/topics/native-vision-transformer-vit
type: topic
---

# Native Vision Transformer (ViT) Overview

The Native Vision Transformer (ViT) is a pure Transformer encoder model, originally devised for natural language processing, adapted for image recognition tasks without convolutional architectures. By tokenizing images into a sequence of fixed-size non-overlapping patches and processing them directly with the standard Transformer encoder architecture, ViT demonstrates that explicit convolutional inductive bias is unnecessary for high-accuracy image classification, provided large-scale pre-training. When pre-trained on extensive datasets such as ImageNet-21k or JFT-300M and transferred to benchmarks including ImageNet, CIFAR-100, and VTAB, ViT attains state-of-the-art results with reduced computational training costs relative to high-capacity convolutional neural networks (CNNs) [2010.11929].

## 1. Patch Embedding and Input Representation

ViT operates by partitioning an input image \(x\in\mathbb{R}^{H\times W\times C}\) into a regular grid of non-overlapping patches of size \(P \times P\), yielding

\[
N = \frac{H\times W}{P^2}
\]

distinct patches. Each patch, \(\mathrm{patch}_i(x)\in\mathbb{R}^{P^2C}\), is linearly projected via \(E \in \mathbb{R}^{(P^2C)\times D}\) into a \(D\)-dimensional embedding:

\[
x_p^i = E \cdot \mathrm{patch}_i(x) \in \mathbb{R}^D.
\]

A learnable classification token \([{\tt CLS}]\in\mathbb{R}^D\) is prepended, and learnable positional encodings \(E_{\text{pos}} \in \mathbb{R}^{(N+1)\times D}\) are added, forming the input sequence:

\[
z_0 = \left[\,[{\tt CLS}];\,x_p^1;\,\dots;\,x_p^N\,\right] + E_\mathrm{pos}.
\]

## 2. Transformer Encoder Architecture

The Transformer encoder processes the \((N+1)\)-length sequence through \(L\) identical layers, each comprising:

- **Pre-LayerNorm**
- **Multi-Head Self-Attention (MSA)** with \(H\) heads:

  \[
  \mathrm{MSA}(X) = [\,\mathrm{head}_1;\dots;\mathrm{head}_H\,] W^O
  \]
  where
  \[
  \mathrm{head}_i = \mathrm{softmax}\left(\frac{Q_i K_i^T}{\sqrt{d_k}}\right) V_i,
  \]
  and \(Q_i = X W^Q_i\), \(K_i = X W^K_i\), \(V_i = X W^V_i\), with \(d_k = D/H\).

- **Residual connection**
- **Pre-LayerNorm**
- **Two-layer MLP**:

  \[
  \mathrm{MLP}(X) = W_2\,\bigl(\mathrm{GELU}(W_1 X)\bigr) + b_2,\quad W_1 \in \mathbb{R}^{rD \times D},\, W_2 \in \mathbb{R}^{D \times rD}
  \]

- **Residual connection**

The stack updates as

\[
\begin{aligned}
z'_\ell & = z_{\ell-1} + \mathrm{MSA}(\mathrm{LN}(z_{\ell-1})) \\
z_\ell & = z'_\ell + \mathrm{MLP}(\mathrm{LN}(z'_\ell))
\end{aligned}
\]

After \(L\) layers, the output associated with the \([{\tt CLS}]\) token is used for image-level classification, following a final LayerNorm and linear classification head.

## 3. Model Variants and Scaling Dimensions

ViT explores the scaling of depth, width, and patch size. The canonical configurations, using the “ViT-X/\(P\)” notation (\(X\in\{\text{B,L,H}\}\) for Base, Large, Huge; \(P\) is patch size), are summarized below.

| Model    | Layers (\(L\)) | Dim (\(D\)) | Heads (\(H\)) | Params | Patch Size (\(P\)) |
|----------|:--------------:|:-----------:|:-------------:|:------:|:------------------:|
| ViT-B/16 |      12        |    768      |      12       |  86M   |        16          |
| ViT-L/16 |      24        |   1024      |      16       | 307M   |        16          |
| ViT-H/14 |      32        |   1280      |      16       | 632M   |        14          |

Smaller patches (lower \(P\)) lead to longer token sequences (\(N\)) and higher spatial resolution, increasing computational cost. Depth (number of layers) is the most effective axis for scaling model capacity, followed by increasing \(D\). Reducing \(P\) also yields accuracy gains without more parameters.

## 4. Training and Transfer Protocols

ViT models are pre-trained on large datasets: ImageNet-21k (14M images, 21k classes) and JFT-300M (303M images, 18k classes, de-duplicated vs. downstream sets). The optimization strategy employs Adam (\(\beta_1 = 0.9, \beta_2 = 0.999\)), weight decay of ≈0.1, and large batch size (4096). Training uses a warm-up period, followed by linear or cosine learning rate decay, and dropout ≈0.1 on MLPs (optionally none for the largest models).

Fine-tuning is performed by discarding the pre-trained classification head and adding a zero-initialized linear layer (\(D \times K\), for \(K\) classes). Typically, fine-tuning operates at higher input resolution (384–512 px), 2D-interpolating the positional embeddings. Fine-tuning uses SGD with momentum 0.9, typical batch sizes of 512, cosine learning rate decay, and no weight decay. Standard hyperparameter grids are applied for learning rate search.

ImageNet fine-tuning occurs over 20,000 steps; smaller target sets use 500–10,000 steps. When pre-trained only on ImageNet (1.3M images), ViT-Large models underperform relative to CNNs. Pre-training on larger datasets (IN-21k, JFT-300M) is critical for strong performance [2010.11929].

## 5. Performance and Computational Characteristics

Empirical evaluations demonstrate:

- **ViT-H/14** achieves 88.55% top-1 on ImageNet, 90.72% on Imagenet-Real, 94.55% on CIFAR-100, and 77.63% on VTAB-19 after JFT-300M pre-training.
- **ViT-L/16** achieves 87.76% on ImageNet, surpassing ResNet152×4 (BiT-L, 87.54%) while utilizing only 0.68k TPUv3-core-days (vs. 9.9k for BiT-L).
- ViT-L/16 pre-trained only on ImageNet-21k achieves 85.30% top-1 in 0.23k core-days.

ViT models require 2–4× fewer pre-training FLOPs than comparably accurate ResNets (R50–R200×3). Hybrid ViT+CNN architectures offer advantages only at smaller compute budgets. Inference speed on TPUv3 is comparable to ResNets, with higher memory efficiency, permitting larger per-core batch sizes [2010.11929].

## 6. Key Insights: Data, Inductive Bias, and Attention

The data scale used for pre-training is critical. On large datasets (14M–300M images), ViT models rapidly surpass CNNs, indicating that scale outweighs explicit convolutional inductive bias. Overfitting arises on small datasets, confirmed by few-shot linear probe experiments.

Model performance correlates more with total pre-training compute than with parameter count. Depth scaling (up to 32 layers, with diminishing returns beyond 64) is especially effective. Visualizations of self-attention show that some heads operate globally from early layers, while others are local; the receptive field size grows systematically with depth. Attention rollout analyses reveal that the \([{\tt CLS}]\)-to-patch attention vector localizes on semantically salient regions.

Positional embeddings are critical: simple learnable 1D embeddings suffice, with no observed benefit for 2D or relative positional embeddings. All positional encoding variants outperform models without positional bias by a wide margin.

Hybrid models, combining a shallow ResNet stem with ViT, are beneficial primarily at low compute but offer no significant advantage when scaling to large data and model sizes.

Preliminary masked-patch prediction (BERT-style) self-supervision achieves 79.9% top-1 for ViT-B/16 on ImageNet (vs. 75.9% from scratch), but remains ∼4% below supervised pre-training [2010.11929].

## 7. Significance and Future Directions

ViT demonstrates that a pure Transformer architecture, with minimal vision-specific inductive bias, can achieve and surpass state-of-the-art accuracy in image classification, provided access to large-scale training data and compute. This finding challenges the necessity of convolutional priors in computer vision, and enables Transformer-only architectures for tasks beyond classification, such as detection, segmentation, and self-supervision. A plausible implication is that Transformer-based architectures, with sufficient scale, may continue to subsume traditionally convolutional approaches as resources and datasets grow [2010.11929].

Source: https://www.emergentmind.com/topics/native-vision-transformer-vit