---
title: 'ViTFS: Efficiency in Vision Transformers'
url: https://www.emergentmind.com/topics/vitfs
type: topic
---

# ViTFS: Efficiency in Vision Transformers

Searching arXiv for the cited papers and closely related uses of “ViTFS” to ground the article.
Searching for "2410.22709 FilterViT and DropoutViT" on arXiv.
Searching arXiv for "Vision Transformers From Scratch teacher-guided data augmentation ViTFS".
ViTFS is not a single uniformly standardized term in the recent arXiv literature. In the materials considered here, it denotes at least three distinct but related ideas: a family of **Vision Transformers From Scratch** for fine-grained image recognition [2507.12157], a **LayerNorm-shift-centric fine-tuning view of Vision Transformer foundation models** [2508.07577], and, by conceptual mapping rather than explicit paper nomenclature, **ViT-style feature selection** through token or frame sparsification in images and videos [2410.22709], [2401.12471]. Across these uses, the common thread is selective efficiency: architectural simplification, parameter-efficient adaptation, or evidence selection under data, compute, or deployment constraints.

## 1. Terminological scope and disambiguation

The label “ViTFS” is attached to different objects across the cited works. One paper explicitly defines it as **Vision Transformers From Scratch** [2507.12157]. Another summary uses **ViTFs** for **Vision Transformer foundation models** and studies fine-tuning through LayerNorm shifts [2508.07577]. In the feature-selection setting, the paper on FilterViT and DropOutVIT does **not** use the term “ViTFS,” but its FilterMobileViT is described as a direct instantiation of Vision Transformer Feature Selection because it computes per-pixel or token saliency, retains the top-$K$ tokens per layer, and performs self-attention only on the selected tokens [2410.22709]. In video, the abstract presents **VidTFS**, while the detailed description identifies the modular framework as **ZERO** and maps the requested dynamic frame selection functionality to ZERO’s Evidence Selector [2401.12471].

| Usage | Source | Core mechanism |
|---|---|---|
| Vision Transformers From Scratch | "Fine-Grained Image Recognition from Scratch with Teacher-Guided Data Augmentation" [2507.12157] | BN-based ViT family trained from random initialization |
| Vision Transformer foundation models + LayerNorm shifts | "Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification" [2508.07577] | Cyclic LayerNorm fine-tuning with $\gamma$ rescaling |
| ViT-style feature selection | "FilterViT and DropoutViT" [2410.22709] | Learned or random top-$K$ token selection for attention |
| Dynamic frame selection in video | "Training-Free Action Recognition and Goal Inference with Dynamic Frame Selection" [2401.12471] | CLIP-based evidence selection over frames |

This multiplicity suggests that “ViTFS” functions more as a family resemblance across efficiency-oriented ViT methods than as a single canonical method name.

## 2. Vision Transformers From Scratch

In "Fine-Grained Image Recognition from Scratch with Teacher-Guided Data Augmentation" [2507.12157], ViTFS stands for **Vision Transformers From Scratch**. It is presented as a family of compact, hardware-aware Vision Transformers trained entirely from random initialization on fine-grained image recognition datasets, without external pretraining. The stated design goals are efficient inference on real devices, data efficiency for fine-grained tasks, and trainability from scratch through a dedicated two-stage framework.

The reported architecture departs from standard pretrained ViTs in several specific ways. LayerNorm is replaced by BatchNorm throughout the network for hardware-friendly inference. The model incorporates a convolutional stem instead of a single linear patch embedding, sinusoidal 2D positional encoding instead of learned positional embeddings, and global average pooling in place of a $[\mathrm{CLS}]$ token. For training stabilization, the design includes LayerScale and register tokens. The provided text explicitly states that token pruning, sparse attention, and specialized low-resolution handling are **not** part of ViTFS in this formulation [2507.12157].

The only explicitly reported member is **ViTFS-T** or **ViTFS-Tiny**. Its parameter count is **5.6M**. Patch size, embedding dimension, depth, head count, MLP ratio, and FLOPs are not provided. This omission is notable because it places the emphasis on the training framework and deployment behavior rather than on a fully enumerated micro-architecture.

On high-resolution fine-grained image recognition at **448×448**, ViTFS-T is reported to achieve **92.2** on FGVC-Aircraft, **94.5** on Stanford Cars, and **86.3** on CUB-200-2011 [2507.12157]. The comparison point in the same report is a pretrained DeiT-B with **86.4M** parameters and ImageNet-1K pretraining, which achieves **90.3**, **92.9**, and **88.0** on the same datasets, respectively. A pretrained ViT-B with **86.4M** parameters and **14,197K** images is reported at **90.0** on Aircraft, **92.5** on Cars, and **90.8** on CUB. The paper’s stated takeaway is that ViTFS-T outperforms pretrained DeiT-B on Aircraft and Cars, is close on CUB, uses **15.3× fewer parameters**, and is trained on orders of magnitude less data [2507.12157].

A further deployment-oriented claim concerns normalization choice. Replacing LayerNorm with BatchNorm is reported to yield a **2.6× inference speedup** and **19% lower peak memory** on a Galaxy A53 smartphone for similar ViT designs [2507.12157]. This makes ViTFS, in this sense, an explicit intervention on both the training regime and the hardware execution path.

## 3. Teacher-Guided Data Augmentation as the enabling training framework

The ViTFS model family in [2507.12157] is enabled by **TGDA**, a two-stage training procedure combining data-aware augmentation and weak supervision through a fine-grained-aware teacher. Stage 1 trains a specialized teacher with a **Part Attention Module (PAM)**. The teacher backbones are described as fine-grained models such as **WS-DAN** or **CAL**, and the reported implementation detail is that **ResNet-101 teachers are fine-tuned for 50 epochs with SGD based on the CAL recipe**. Whether the teacher starts from pretrained weights is explicitly stated as not provided.

PAM produces semantically meaningful part attention maps that drive two augmentations. **Attention cropping** zooms into discriminative regions, while **attention dropping** masks those regions to force learning of complementary cues. These attention-guided views are then used to train the student ViTFS model from random initialization under knowledge distillation.

The student objective combines supervised learning and distillation. With student logits $z_s$, teacher logits $z_t$, labels $y$, original image $I_{\mathrm{org}}$, augmented images $I_{\mathrm{aug}}$, and temperature $T$, the reported losses are
$$
L_{\mathrm{CE}} = - \sum_c y_c \log \mathrm{softmax}(z_s(I_{\mathrm{org}}))_c,
$$
$$
L_{\mathrm{KD,org}} = \mathrm{KL}(\mathrm{softmax}_T(z_t(I_{\mathrm{org}})) \,\|\, \mathrm{softmax}_T(z_s(I_{\mathrm{org}}))),
$$
$$
L_{\mathrm{KD,aug}} = \mathrm{KL}(\mathrm{softmax}_T(z_t(I_{\mathrm{aug}})) \,\|\, \mathrm{softmax}_T(z_s(I_{\mathrm{aug}}))),
$$
and
$$
L_{\mathrm{total}} = \alpha \cdot L_{\mathrm{CE}} + \beta \cdot [L_{\mathrm{KD,org}} + L_{\mathrm{KD,aug}}].
$$
The reported hyperparameters are **$\beta = 10$** and **$T = 7$**; the value of $\alpha$ is not explicitly provided [2507.12157]. Label smoothing and stochastic depth are used, and the student is trained with **AdamW** for **800 epochs**. Learning-rate schedule, base learning rate, batch size, weight decay, warmup, and gradient clipping are not specified.

The algorithmic consequence is that ViTFS is not merely a small ViT architecture. It is a training recipe that makes from-scratch transformer optimization viable under target-dataset-only supervision. This suggests that, in this usage, the central contribution lies at the intersection of architecture, augmentation, and teacher-guided optimization rather than in a narrowly defined transformer block innovation.

## 4. LayerNorm shifts and ViTF fine-tuning

A separate use of the term appears in "Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification" [2508.07577]. Here the core object is the **Vision Transformer foundation model**, and the study focuses on how **LayerNorm shifts** encode source-to-target domain transitions under scarce data and domain shift. The paper treats LayerNorm as a privileged adaptation locus because it controls feature statistics at every block.

For an input vector $x$, LayerNorm is written as
$$
y = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} \odot \gamma + \beta.
$$
If $\gamma_{\mathrm{pre}}, \beta_{\mathrm{pre}}$ are pretrained parameters and $\gamma_{\mathrm{ft}}, \beta_{\mathrm{ft}}$ are post-fine-tuning parameters, the per-layer shifts are
$$
\Delta \gamma_i = \gamma_{i,\mathrm{ft}} - \gamma_{i,\mathrm{pre}}, \qquad
\Delta \beta_i = \beta_{i,\mathrm{ft}} - \beta_{i,\mathrm{pre}}.
$$
The global LayerNorm shift is aggregated as
$$
\|LN^T - LN^S\|_G := \sum_{i=1}^n \left( \|\gamma_i^T - \gamma_i^S\|_2 + \|\beta_i^T - \beta_i^S\|_2 \right).
$$

The central scalar in the paper is the **Fine-tuning Shift Ratio (FSR)**,
$$
FSR := \frac{\mathrm{Shift}_{\mathrm{data}}(X^S, X^T)}{\mathrm{Shift}_{\mathrm{data}}(X^S, X^{T*}) + \epsilon},
$$
where the data shift metric is the Wasserstein distance. The interpretation given is direct: if $FSR \approx 1$, the labeled target training data is representative of the full target domain; if $FSR < 1$, it under-represents the target-domain shift; if $FSR > 1$, it over-represents it [2508.07577].

To correct under- or over-shoot in learned LayerNorm shifts, the paper proposes rescaling only the LayerNorm scale parameter:
$$
\gamma_{i,\mathrm{aligned}} := \lambda \cdot \gamma_{i,\mathrm{ft}}, \qquad
\beta_{i,\mathrm{aligned}} := \beta_{i,\mathrm{ft}}.
$$
The reported empirical regularity is that the optimal $\lambda$ is **negatively correlated** with $FSR$: lower $FSR$ tends to favor larger $\lambda$, whereas higher $FSR$ tends to favor smaller $\lambda$ or values near $1$ [2508.07577]. The paper explicitly recommends avoiding $\beta$ scaling and states that pathology ViTFs often prefer **$\lambda \le 1$**.

This rescaling is embedded in a **cyclic fine-tuning framework**. Each cycle alternates between a predictor-only stage, in which the backbone is frozen and the linear predictor is trained, and a LayerNorm-only stage, in which all layers except LayerNorm are frozen and only LayerNorm parameters are adapted. After the LayerNorm stage, $\gamma$ is rescaled by $\lambda$. This is presented as a stabilization strategy under scarce data and as a way to keep the interpretation of LayerNorm shifts coherent.

The empirical evidence spans **DomainNet**, **SUN**, **DTD**, and pathology datasets including **Bach**, **Breakhis**, **CCRCC**, and **Glas** [2508.07577]. On DomainNet with MAE, the averages reported are **69.58** for ID A→A, **57.26** for OOD A→B, **61.50** for ID B→B, and **63.39** for OOD B→A, each exceeding or matching the reported LP+LN, LP, and LP+FM baselines. On toy Gaussian experiments, rescaling improves accuracy in **1326** of **1815** cases, or approximately **73%**, with average gain **2.39%**, and more than **90%** of cases show non-decreasing accuracy. The paper states that OOD tasks tend to yield lower $FSR$ and higher $\lambda$, whereas pathology ViTFs behave more like ID settings and favor conservative updates.

In this sense, ViTFS refers not to from-scratch architecture design, but to a parameter-efficient fine-tuning doctrine: adaptation should be concentrated in normalization parameters, interpreted through dataset representativeness, and corrected through post hoc scaling.

## 5. Token-level feature selection in images

A third usage is conceptual rather than nominal. "FilterViT and DropoutViT" does **not** use the term “ViTFS,” but the provided description states that **FilterMobileViT is a direct instantiation of Vision Transformer Feature Selection** [2410.22709]. The mechanism is explicit: learn a saliency map over pixels or tokens, keep only the top-$K$ positions per layer, perform attention only on the selected subset, and then write the attended tokens back to the spatial map.

The feature extractor produces a feature map
$$
X_f \in \mathbb{R}^{B \times C \times H \times W}.
$$
A convolutional head generates a saliency map
$$
M = \sigma(\mathrm{Conv}(x)) \in \mathbb{R}^{B \times 1 \times H \times W},
$$
or implicitly, for a pixel feature $x_i \in \mathbb{R}^C$,
$$
s_i = \sigma(w^T x_i + b).
$$
Selection is then performed by hard top-$K$ indexing:
$$
\mathrm{indices} = \mathrm{TopK}(\mathrm{imp}, K), \qquad
M_i = 1 \text{ if } i \in \mathrm{TopK}(s, K), \text{ else } 0.
$$
The model applies soft masking
$$
X_{\mathrm{masked}} = X_f \odot s,
$$
gathers selected tokens with positional encoding,
$$
X' = \{ X_f[i] + \mathrm{pos}_i \mid M_i = 1 \},
$$
runs standard attention on the selected set,
$$
Q = X'W_Q,\quad K = X'W_K,\quad V = X'W_V,
$$
$$
A = \mathrm{softmax}(QK^T/\sqrt{d_k}),\quad Y = AV,
$$
and writes back the outputs to the selected spatial positions.

The complexity claim is the standard reduction from
$$
C_{\mathrm{attn}} = O(N^2 d_k)
$$
to
$$
C'_{\mathrm{attn}} = O(K^2 d_k),
$$
with ideal attention-core speed-up approximately $(N/K)^2$ when $K \ll N = H \times W$ [2410.22709]. The text is explicit, however, that end-to-end latency depends on gather/scatter overhead, the convolutional backbone, and the platform.

On the first ImageNet-100 subset at **224×224**, the reported numbers are **0.861** accuracy and **1.89M** parameters for FilterMobileViT, versus **0.851** and **5.00M** for MobileViT, and **0.876** and **10.59M** for Tiny-ViT [2410.22709]. Throughput is reported as **410 FPS** on CUDA RTX 4090 and **5.60 FPS** on CPU M1 Pro for FilterMobileViT, compared with **485** and **6.14** for MobileViT. The paper therefore states that FilterMobileViT is more parameter-efficient and more accurate than MobileViT, but **not faster in overall FPS on CUDA or CPU** in the reported setup.

Interpretability is a central part of the formulation. The filter masks are visualized across layers: **Layer 1** shows broader focus including edges and general features, **Layer 2** tightens on the main object, and **Layer 3** expands to background or context again. The paper explicitly notes that these masks provide human-interpretable evidence without extra procedures such as Attention Rollout or Grad-CAM [2410.22709].

The stochastic ablation is **DropOutVIT**, which replaces deterministic top-$K$ selection with random token sampling. It operates as structured token-level dropout. The reported empirical pattern is that DropOutVIT matches or slightly outperforms FilterMobileViT early in training, but plateaus around **epoch $\approx 50$**, whereas FilterMobileViT continues improving, presumably because learned saliency becomes more informative. Several limitations are also stated: hard top-$K$ is non-differentiable, the per-layer $K_l$ schedule is unspecified, gather/scatter can erase theoretical speed-ups, and robustness beyond ImageNet-100 subsets is not studied.

## 6. Dynamic frame selection in videos

The video counterpart extends the feature-selection logic from tokens to frames. In "Training-Free Action Recognition and Goal Inference with Dynamic Frame Selection" [2401.12471], the abstract presents **VidTFS**, while the detailed description explains the framework as **ZERO**. The core mechanism is the same kind of selective evidence reduction: only the most relevant frames from an untrimmed video are re-captioned and re-reasoned over.

ZERO is organized into **Perception**, **Assumption**, and **Identification** stages. The input video is
$$
V = \{x_1, x_2, \dots, x_N\}.
$$
A frozen captioning model, **BLIP-2 with FLAN-T5-XXL**, generates captions for an initial subset of frames. A frozen LLM, **Vicuna-13B** or **GPT-3.5**, generates top-$K$ hypotheses and textual scripts for each hypothesis. A frozen **CLIP ViT-B/16** encoder then aligns all sampled frames to the generated steps.

If $I_i = F_{\mathrm{VIS}}(x_i)$ is the frame embedding and $T_j = F_{\mathrm{TXT}}(s_j)$ is the embedding of a hypothesized step, the step-frame similarity is
$$
s(x_i, s_j) = \frac{I_i \cdot T_j}{\|I_i\|\,\|T_j\|}.
$$
The resulting similarity matrix $S \in \mathbb{R}^{M \times N}$ is flattened, and the framework selects the **top-$M$ unique frames** with the highest scores. These selected frames are re-captioned, a refined description is formed, and the LLM performs a second inference pass before a final choice is made over the union of the initial and refined hypothesis sets [2401.12471].

The pipeline is explicitly **training-free** and **open-vocabulary** because BLIP-2, CLIP, and the LLM are all frozen. The text also notes that the selector adapts per video because the step texts are generated from that video’s own captions rather than from a fixed template.

The reported evaluation spans **CrossTask**, **COIN**, **UCF101**, and **ActivityNet**. For open-ended goal inference on CrossTask with GPT-3.5, the paper reports **CIDEr 1.186** at observation ratio $\rho=0.5$, **SPICE 0.234**, and **SBERT 0.549**. In close-ended goal inference, ZERO with GPT-3.5 reaches **76.5%** Top-1 on CrossTask at $\rho=0.5$, exceeding reported video classification baselines such as ViFi-CLIP at **67.9%**, X-CLIP at **62.2%**, and MAXI at **66.0%**. For close-ended action recognition, the reported Top-1 is **66.3%** on UCF101 and **41.1%** on ActivityNet with Vicuna-13B [2401.12471].

The ablation evidence is direct. Removing the Evidence Selector degrades performance substantially; for CrossTask at $\rho=0.3$, the reported CIDEr is **0.845** with the selector, **0.664** with SBERT-based text-only selection, and **0.394** without selection. The number of selected frames also matters: **$M=16$** is reported as optimal on CrossTask, with **$M=8$** and **$M=32$** performing slightly worse.

The main limitations are temporal. The framework does not explicitly model temporal order beyond LLM scripts and concatenation with “then,” so long-range temporal dependencies and subtle motion cues can be missed. This suggests that, in the video setting, “ViTFS” or “VidTFS” is less a transformer architecture than a training-free evidence selection regime built around frozen multimodal models.

## 7. Related directions: token fusion, training-free search, and recurring limitations

Two additional lines of work help situate ViTFS within the broader space of efficiency-oriented ViT methods.

The first is **token fusion**. "Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers" introduces **MCTF**, which reduces the number of tokens by fusing rather than pruning them [2403.10030]. The fusion score combines similarity, informativeness from one-step-ahead attention, and token size:
$$
W(x_i, x_j) = \prod_k (W^k(x_i, x_j))^{\tau_k},
$$
with default temperatures $[\tau_{\mathrm{info}}, \tau_{\mathrm{sim}}, \tau_{\mathrm{size}}] = [1, 1/20, 1/40]$. The fused token uses weighted averaging,
$$
\delta(\{x_i\}) = \frac{\sum a_i s_i x_i}{\sum a_i s_i}.
$$
On ImageNet-1K, the reported result is about **44%** FLOPs reduction for DeiT-T and DeiT-S together with accuracy gains of **+0.5%** and **+0.3%**, respectively. The paper is not a ViTFS paper in name, but it occupies the same design space as token selection: reduce the quadratic attention cost while preserving useful information [2403.10030].

The second is **training-free search**. "Training-free Transformer Architecture Search" proposes **TF-TAS**, which ranks ViT candidates without training by a **DSS-indicator** composed of MSA synaptic diversity and MLP synaptic saliency [2203.12217]. The architecture-level score is
$$
SDSS(\mathcal{A}) = \sum_l D^l_{\mathrm{MSA}} + \sum_k S^k_{\mathrm{MLP}}.
$$
The reported search cost drops from about **24 GPU days** to **less than 0.5 GPU days**, and the resulting TF-TAS-Ti reaches **75.3%** Top-1 on ImageNet with **5.9M** parameters and **1.4B** FLOPs [2203.12217]. This is another neighboring interpretation of “ViTFS” as training-free optimization of ViT design rather than of features or normalization parameters.

Across these usages, several recurring limitations are explicit in the source material. FilterMobileViT leaves the $K_l$ schedule unspecified and provides no theoretical guarantees [2410.22709]. The LayerNorm-shift framework requires either direct or proxy reasoning about target-domain representativeness, and direct $FSR$ computation is often infeasible in practice [2508.07577]. ViTFS from scratch leaves many micro-architectural and optimization hyperparameters unspecified [2507.12157]. The video selection framework remains weak in explicit temporal modeling [2401.12471]. These constraints indicate that “ViTFS” is best understood not as a single finalized method family, but as an active cluster of approaches that use selectivity—of parameters, tokens, frames, or architectures—to make Vision Transformers more data-efficient, compute-efficient, or deployment-efficient.

Source: https://www.emergentmind.com/topics/vitfs