ViTFS: Efficiency in Vision Transformers

Updated 6 July 2026

ViTFS is a versatile term that covers efficiency methods in Vision Transformers, including training from scratch, fine-tuning via LayerNorm shifts, and token/frame feature selection.
The approaches target computational overhead reduction and parameter efficiency, as shown by improved inference speeds and competitive performance on fine-grained image and video recognition benchmarks.
By employing techniques like teacher-guided augmentation, cyclic fine-tuning, and selective evidence reduction, ViTFS offers practical strategies for optimizing transformer deployment on resource-constrained devices.

Searching arXiv for the cited papers and closely related uses of “ViTFS” to ground the article. Searching for "(Sun, 2024) FilterViT and DropoutViT" on arXiv. Searching arXiv for "Vision Transformers From Scratch teacher-guided data augmentation ViTFS". ViTFS is not a single uniformly standardized term in the recent arXiv literature. In the materials considered here, it denotes at least three distinct but related ideas: a family of Vision Transformers From Scratch for fine-grained image recognition (Rios et al., 16 Jul 2025), a LayerNorm-shift-centric fine-tuning view of Vision Transformer foundation models (Tan et al., 11 Aug 2025), and, by conceptual mapping rather than explicit paper nomenclature, ViT-style feature selection through token or frame sparsification in images and videos (Sun, 2024, Keat et al., 2024). Across these uses, the common thread is selective efficiency: architectural simplification, parameter-efficient adaptation, or evidence selection under data, compute, or deployment constraints.

1. Terminological scope and disambiguation

The label “ViTFS” is attached to different objects across the cited works. One paper explicitly defines it as Vision Transformers From Scratch (Rios et al., 16 Jul 2025). Another summary uses ViTFs for Vision Transformer foundation models and studies fine-tuning through LayerNorm shifts (Tan et al., 11 Aug 2025). In the feature-selection setting, the paper on FilterViT and DropOutVIT does not use the term “ViTFS,” but its FilterMobileViT is described as a direct instantiation of Vision Transformer Feature Selection because it computes per-pixel or token saliency, retains the top- $K$ tokens per layer, and performs self-attention only on the selected tokens (Sun, 2024). In video, the abstract presents VidTFS, while the detailed description identifies the modular framework as ZERO and maps the requested dynamic frame selection functionality to ZERO’s Evidence Selector (Keat et al., 2024).

Usage	Source	Core mechanism
Vision Transformers From Scratch	"Fine-Grained Image Recognition from Scratch with Teacher-Guided Data Augmentation" (Rios et al., 16 Jul 2025)	BN-based ViT family trained from random initialization
Vision Transformer foundation models + LayerNorm shifts	"Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification" (Tan et al., 11 Aug 2025)	Cyclic LayerNorm fine-tuning with $\gamma$ rescaling
ViT-style feature selection	"FilterViT and DropoutViT" (Sun, 2024)	Learned or random top- $K$ token selection for attention
Dynamic frame selection in video	"Training-Free Action Recognition and Goal Inference with Dynamic Frame Selection" (Keat et al., 2024)	CLIP-based evidence selection over frames

This multiplicity suggests that “ViTFS” functions more as a family resemblance across efficiency-oriented ViT methods than as a single canonical method name.

2. Vision Transformers From Scratch

In "Fine-Grained Image Recognition from Scratch with Teacher-Guided Data Augmentation" (Rios et al., 16 Jul 2025), ViTFS stands for Vision Transformers From Scratch. It is presented as a family of compact, hardware-aware Vision Transformers trained entirely from random initialization on fine-grained image recognition datasets, without external pretraining. The stated design goals are efficient inference on real devices, data efficiency for fine-grained tasks, and trainability from scratch through a dedicated two-stage framework.

The reported architecture departs from standard pretrained ViTs in several specific ways. LayerNorm is replaced by BatchNorm throughout the network for hardware-friendly inference. The model incorporates a convolutional stem instead of a single linear patch embedding, sinusoidal 2D positional encoding instead of learned positional embeddings, and global average pooling in place of a $[\mathrm{CLS}]$ token. For training stabilization, the design includes LayerScale and register tokens. The provided text explicitly states that token pruning, sparse attention, and specialized low-resolution handling are not part of ViTFS in this formulation (Rios et al., 16 Jul 2025).

The only explicitly reported member is ViTFS-T or ViTFS-Tiny. Its parameter count is 5.6M. Patch size, embedding dimension, depth, head count, MLP ratio, and FLOPs are not provided. This omission is notable because it places the emphasis on the training framework and deployment behavior rather than on a fully enumerated micro-architecture.

On high-resolution fine-grained image recognition at 448×448, ViTFS-T is reported to achieve 92.2 on FGVC-Aircraft, 94.5 on Stanford Cars, and 86.3 on CUB-200-2011 (Rios et al., 16 Jul 2025). The comparison point in the same report is a pretrained DeiT-B with 86.4M parameters and ImageNet-1K pretraining, which achieves 90.3, 92.9, and 88.0 on the same datasets, respectively. A pretrained ViT-B with 86.4M parameters and 14,197K images is reported at 90.0 on Aircraft, 92.5 on Cars, and 90.8 on CUB. The paper’s stated takeaway is that ViTFS-T outperforms pretrained DeiT-B on Aircraft and Cars, is close on CUB, uses 15.3× fewer parameters, and is trained on orders of magnitude less data (Rios et al., 16 Jul 2025).

A further deployment-oriented claim concerns normalization choice. Replacing LayerNorm with BatchNorm is reported to yield a 2.6× inference speedup and 19% lower peak memory on a Galaxy A53 smartphone for similar ViT designs (Rios et al., 16 Jul 2025). This makes ViTFS, in this sense, an explicit intervention on both the training regime and the hardware execution path.

3. Teacher-Guided Data Augmentation as the enabling training framework

The ViTFS model family in (Rios et al., 16 Jul 2025) is enabled by TGDA, a two-stage training procedure combining data-aware augmentation and weak supervision through a fine-grained-aware teacher. Stage 1 trains a specialized teacher with a Part Attention Module (PAM). The teacher backbones are described as fine-grained models such as WS-DAN or CAL, and the reported implementation detail is that ResNet-101 teachers are fine-tuned for 50 epochs with SGD based on the CAL recipe. Whether the teacher starts from pretrained weights is explicitly stated as not provided.

PAM produces semantically meaningful part attention maps that drive two augmentations. Attention cropping zooms into discriminative regions, while attention dropping masks those regions to force learning of complementary cues. These attention-guided views are then used to train the student ViTFS model from random initialization under knowledge distillation.

The student objective combines supervised learning and distillation. With student logits $z_s$ , teacher logits $z_t$ , labels $y$ , original image $I_{\mathrm{org}}$ , augmented images $I_{\mathrm{aug}}$ , and temperature $T$ , the reported losses are

$\gamma$ 0

$\gamma$ 1

$\gamma$ 2

and

$\gamma$ 3

The reported hyperparameters are $\gamma$ 4 and $\gamma$ 5; the value of $\gamma$ 6 is not explicitly provided (Rios et al., 16 Jul 2025). Label smoothing and stochastic depth are used, and the student is trained with AdamW for 800 epochs. Learning-rate schedule, base learning rate, batch size, weight decay, warmup, and gradient clipping are not specified.

The algorithmic consequence is that ViTFS is not merely a small ViT architecture. It is a training recipe that makes from-scratch transformer optimization viable under target-dataset-only supervision. This suggests that, in this usage, the central contribution lies at the intersection of architecture, augmentation, and teacher-guided optimization rather than in a narrowly defined transformer block innovation.

4. LayerNorm shifts and ViTF fine-tuning

A separate use of the term appears in "Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification" (Tan et al., 11 Aug 2025). Here the core object is the Vision Transformer foundation model, and the study focuses on how LayerNorm shifts encode source-to-target domain transitions under scarce data and domain shift. The paper treats LayerNorm as a privileged adaptation locus because it controls feature statistics at every block.

For an input vector $\gamma$ 7, LayerNorm is written as

$\gamma$ 8

If $\gamma$ 9 are pretrained parameters and $K$ 0 are post-fine-tuning parameters, the per-layer shifts are

$K$ 1

The global LayerNorm shift is aggregated as

$K$ 2

The central scalar in the paper is the Fine-tuning Shift Ratio (FSR),

$K$ 3

where the data shift metric is the Wasserstein distance. The interpretation given is direct: if $K$ 4, the labeled target training data is representative of the full target domain; if $K$ 5, it under-represents the target-domain shift; if $K$ 6, it over-represents it (Tan et al., 11 Aug 2025).

To correct under- or over-shoot in learned LayerNorm shifts, the paper proposes rescaling only the LayerNorm scale parameter:

$K$ 7

The reported empirical regularity is that the optimal $K$ 8 is negatively correlated with $K$ 9: lower $[\mathrm{CLS}]$ 0 tends to favor larger $[\mathrm{CLS}]$ 1, whereas higher $[\mathrm{CLS}]$ 2 tends to favor smaller $[\mathrm{CLS}]$ 3 or values near $[\mathrm{CLS}]$ 4 (Tan et al., 11 Aug 2025). The paper explicitly recommends avoiding $[\mathrm{CLS}]$ 5 scaling and states that pathology ViTFs often prefer $[\mathrm{CLS}]$ 6.

This rescaling is embedded in a cyclic fine-tuning framework. Each cycle alternates between a predictor-only stage, in which the backbone is frozen and the linear predictor is trained, and a LayerNorm-only stage, in which all layers except LayerNorm are frozen and only LayerNorm parameters are adapted. After the LayerNorm stage, $[\mathrm{CLS}]$ 7 is rescaled by $[\mathrm{CLS}]$ 8. This is presented as a stabilization strategy under scarce data and as a way to keep the interpretation of LayerNorm shifts coherent.

The empirical evidence spans DomainNet, SUN, DTD, and pathology datasets including Bach, Breakhis, CCRCC, and Glas (Tan et al., 11 Aug 2025). On DomainNet with MAE, the averages reported are 69.58 for ID A→A, 57.26 for OOD A→B, 61.50 for ID B→B, and 63.39 for OOD B→A, each exceeding or matching the reported LP+LN, LP, and LP+FM baselines. On toy Gaussian experiments, rescaling improves accuracy in 1326 of 1815 cases, or approximately 73%, with average gain 2.39%, and more than 90% of cases show non-decreasing accuracy. The paper states that OOD tasks tend to yield lower $[\mathrm{CLS}]$ 9 and higher $z_s$ 0, whereas pathology ViTFs behave more like ID settings and favor conservative updates.

In this sense, ViTFS refers not to from-scratch architecture design, but to a parameter-efficient fine-tuning doctrine: adaptation should be concentrated in normalization parameters, interpreted through dataset representativeness, and corrected through post hoc scaling.

5. Token-level feature selection in images

A third usage is conceptual rather than nominal. "FilterViT and DropoutViT" does not use the term “ViTFS,” but the provided description states that FilterMobileViT is a direct instantiation of Vision Transformer Feature Selection (Sun, 2024). The mechanism is explicit: learn a saliency map over pixels or tokens, keep only the top- $z_s$ 1 positions per layer, perform attention only on the selected subset, and then write the attended tokens back to the spatial map.

The feature extractor produces a feature map

$z_s$ 2

A convolutional head generates a saliency map

$z_s$ 3

or implicitly, for a pixel feature $z_s$ 4,

$z_s$ 5

Selection is then performed by hard top- $z_s$ 6 indexing:

$z_s$ 7

The model applies soft masking

$z_s$ 8

gathers selected tokens with positional encoding,

$z_s$ 9

runs standard attention on the selected set,

$z_t$ 0

$z_t$ 1

and writes back the outputs to the selected spatial positions.

The complexity claim is the standard reduction from

$z_t$ 2

$z_t$ 3

with ideal attention-core speed-up approximately $z_t$ 4 when $z_t$ 5 (Sun, 2024). The text is explicit, however, that end-to-end latency depends on gather/scatter overhead, the convolutional backbone, and the platform.

On the first ImageNet-100 subset at 224×224, the reported numbers are 0.861 accuracy and 1.89M parameters for FilterMobileViT, versus 0.851 and 5.00M for MobileViT, and 0.876 and 10.59M for Tiny-ViT (Sun, 2024). Throughput is reported as 410 FPS on CUDA RTX 4090 and 5.60 FPS on CPU M1 Pro for FilterMobileViT, compared with 485 and 6.14 for MobileViT. The paper therefore states that FilterMobileViT is more parameter-efficient and more accurate than MobileViT, but not faster in overall FPS on CUDA or CPU in the reported setup.

Interpretability is a central part of the formulation. The filter masks are visualized across layers: Layer 1 shows broader focus including edges and general features, Layer 2 tightens on the main object, and Layer 3 expands to background or context again. The paper explicitly notes that these masks provide human-interpretable evidence without extra procedures such as Attention Rollout or Grad-CAM (Sun, 2024).

The stochastic ablation is DropOutVIT, which replaces deterministic top- $z_t$ 6 selection with random token sampling. It operates as structured token-level dropout. The reported empirical pattern is that DropOutVIT matches or slightly outperforms FilterMobileViT early in training, but plateaus around epoch $z_t$ 7, whereas FilterMobileViT continues improving, presumably because learned saliency becomes more informative. Several limitations are also stated: hard top- $z_t$ 8 is non-differentiable, the per-layer $z_t$ 9 schedule is unspecified, gather/scatter can erase theoretical speed-ups, and robustness beyond ImageNet-100 subsets is not studied.

6. Dynamic frame selection in videos

The video counterpart extends the feature-selection logic from tokens to frames. In "Training-Free Action Recognition and Goal Inference with Dynamic Frame Selection" (Keat et al., 2024), the abstract presents VidTFS, while the detailed description explains the framework as ZERO. The core mechanism is the same kind of selective evidence reduction: only the most relevant frames from an untrimmed video are re-captioned and re-reasoned over.

ZERO is organized into Perception, Assumption, and Identification stages. The input video is

$y$ 0

A frozen captioning model, BLIP-2 with FLAN-T5-XXL, generates captions for an initial subset of frames. A frozen LLM, Vicuna-13B or GPT-3.5, generates top- $y$ 1 hypotheses and textual scripts for each hypothesis. A frozen CLIP ViT-B/16 encoder then aligns all sampled frames to the generated steps.

If $y$ 2 is the frame embedding and $y$ 3 is the embedding of a hypothesized step, the step-frame similarity is

$y$ 4

The resulting similarity matrix $y$ 5 is flattened, and the framework selects the top- $y$ 6 unique frames with the highest scores. These selected frames are re-captioned, a refined description is formed, and the LLM performs a second inference pass before a final choice is made over the union of the initial and refined hypothesis sets (Keat et al., 2024).

The pipeline is explicitly training-free and open-vocabulary because BLIP-2, CLIP, and the LLM are all frozen. The text also notes that the selector adapts per video because the step texts are generated from that video’s own captions rather than from a fixed template.

The reported evaluation spans CrossTask, COIN, UCF101, and ActivityNet. For open-ended goal inference on CrossTask with GPT-3.5, the paper reports CIDEr 1.186 at observation ratio $y$ 7, SPICE 0.234, and SBERT 0.549. In close-ended goal inference, ZERO with GPT-3.5 reaches 76.5% Top-1 on CrossTask at $y$ 8, exceeding reported video classification baselines such as ViFi-CLIP at 67.9%, X-CLIP at 62.2%, and MAXI at 66.0%. For close-ended action recognition, the reported Top-1 is 66.3% on UCF101 and 41.1% on ActivityNet with Vicuna-13B (Keat et al., 2024).

The ablation evidence is direct. Removing the Evidence Selector degrades performance substantially; for CrossTask at $y$ 9, the reported CIDEr is 0.845 with the selector, 0.664 with SBERT-based text-only selection, and 0.394 without selection. The number of selected frames also matters: $I_{\mathrm{org}}$ 0 is reported as optimal on CrossTask, with $I_{\mathrm{org}}$ 1 and $I_{\mathrm{org}}$ 2 performing slightly worse.

The main limitations are temporal. The framework does not explicitly model temporal order beyond LLM scripts and concatenation with “then,” so long-range temporal dependencies and subtle motion cues can be missed. This suggests that, in the video setting, “ViTFS” or “VidTFS” is less a transformer architecture than a training-free evidence selection regime built around frozen multimodal models.

Two additional lines of work help situate ViTFS within the broader space of efficiency-oriented ViT methods.

The first is token fusion. "Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers" introduces MCTF, which reduces the number of tokens by fusing rather than pruning them (Lee et al., 2024). The fusion score combines similarity, informativeness from one-step-ahead attention, and token size:

$I_{\mathrm{org}}$ 3

with default temperatures $I_{\mathrm{org}}$ 4. The fused token uses weighted averaging,

$I_{\mathrm{org}}$ 5

On ImageNet-1K, the reported result is about 44% FLOPs reduction for DeiT-T and DeiT-S together with accuracy gains of +0.5% and +0.3%, respectively. The paper is not a ViTFS paper in name, but it occupies the same design space as token selection: reduce the quadratic attention cost while preserving useful information (Lee et al., 2024).

The second is training-free search. "Training-free Transformer Architecture Search" proposes TF-TAS, which ranks ViT candidates without training by a DSS-indicator composed of MSA synaptic diversity and MLP synaptic saliency (Zhou et al., 2022). The architecture-level score is

$I_{\mathrm{org}}$ 6

The reported search cost drops from about 24 GPU days to less than 0.5 GPU days, and the resulting TF-TAS-Ti reaches 75.3% Top-1 on ImageNet with 5.9M parameters and 1.4B FLOPs (Zhou et al., 2022). This is another neighboring interpretation of “ViTFS” as training-free optimization of ViT design rather than of features or normalization parameters.

Across these usages, several recurring limitations are explicit in the source material. FilterMobileViT leaves the $I_{\mathrm{org}}$ 7 schedule unspecified and provides no theoretical guarantees (Sun, 2024). The LayerNorm-shift framework requires either direct or proxy reasoning about target-domain representativeness, and direct $I_{\mathrm{org}}$ 8 computation is often infeasible in practice (Tan et al., 11 Aug 2025). ViTFS from scratch leaves many micro-architectural and optimization hyperparameters unspecified (Rios et al., 16 Jul 2025). The video selection framework remains weak in explicit temporal modeling (Keat et al., 2024). These constraints indicate that “ViTFS” is best understood not as a single finalized method family, but as an active cluster of approaches that use selectivity—of parameters, tokens, frames, or architectures—to make Vision Transformers more data-efficient, compute-efficient, or deployment-efficient.