TinyDrop: Efficient Token Dropping for ViTs

• TinyDrop is a plug-and-play token dropping framework for ViTs that uses a lightweight guidance model to adaptively retain the most informative tokens.
• It leverages Grad-CAM-derived saliency and a confidence-driven early exit mechanism to achieve significant FLOP reduction (up to 80%–85%) while keeping accuracy nearly intact.
• With its training-free design that requires no modifications or retraining of pre-trained backbones, TinyDrop offers a practical approach for efficient ImageNet classification.

TinyDrop is a tiny-model-guided, training-free, plug-and-play token dropping framework for Vision Transformers (ViTs) that reduces inference cost by using a separate lightweight vision model to decide both whether to terminate inference early and, if the large ViT is still invoked, which image tokens should be retained for attention computation (Wang et al., 3 Sep 2025). It is designed for image classification with frozen pre-trained backbones, requires no architectural modification or retraining of the target ViT, and is evaluated on standard ImageNet-1K classification settings. The central objective is to reduce the cost of processing all patch tokens in large ViTs while keeping accuracy degradation minimal; reported results include up to roughly $80\%$–$85\%$ FLOPs reduction while keeping ImageNet accuracy within $\le 1\%$ of the original model (Wang et al., 3 Sep 2025).

1. Computational setting and design rationale

TinyDrop is motivated by the tokenized structure of standard ViTs. For an image of size $224 \times 224$ and patch size $16 \times 16$, the model processes $T = 14 \times 14 = 196$ patch tokens plus a class token. Because self-attention in each transformer block has quadratic complexity in the sequence length, FLOPs scale roughly with $\mathcal{O}(T^2)$ across layers. Many of these tokens correspond to low-information patches such as uniform background, so a substantial portion of the computation may be expended on tokens with limited utility for classification (Wang et al., 3 Sep 2025).

The framework is positioned against two families of existing acceleration methods. Training-based token pruning or sampling methods, such as DiffRate and ATS, learn per-layer keep rates but require fine-tuning and architectural changes. Training-free methods such as ToMe, Zero-TP, and PaPr avoid some retraining costs, but the description of prior work in TinyDrop identifies several recurrent constraints: fixed or global keep ratios, limited sample adaptivity, sensitivity to architecture-specific heuristics, or incomplete accounting of total overhead. PaPr is singled out as using a tiny ConvNet for patch importance prediction but still relying on a fixed keep rate and being mainly evaluated on smaller backbones (Wang et al., 3 Sep 2025).

TinyDrop replaces those constraints with an independent guidance model $\mathcal{M}_g$, which is a lightweight image classifier that simultaneously produces a confidence score and a saliency map. The confidence determines whether the image can be classified directly by the small model or whether the large target ViT $\mathcal{M}_t$ should be used. If the large model is used, the same confidence drives an adaptive token drop ratio, and the saliency map determines which patch tokens are preserved. This yields a per-sample adaptive schedule rather than a global static keep ratio. A plausible implication is that the framework shifts token-importance estimation away from internal attention heuristics of the large backbone and toward a separate model with its own calibrated predictive signal.

2. Inference pipeline and mathematical formulation

The TinyDrop pipeline begins with inference by the guidance model $\mathcal{M}_g$. Given class prediction probabilities $85\%$0, the confidence is defined as

$85\%$1

If $85\%$2, where $85\%$3 is a confidence threshold, the guidance model’s prediction is used as the final output and the large ViT is skipped entirely. If $85\%$4, the system proceeds to token dropping (Wang et al., 3 Sep 2025).

Token scoring is derived from Grad-CAM on the final block of the guidance model. The resulting saliency map $85\%$5 is resampled to a $85\%$6 grid, such as $85\%$7, and then flattened to $85\%$8, where each element represents the estimated importance of a ViT patch token. The same confidence value $85\%$9 is mapped to a token drop ratio $\le 1\%$0 through

$\le 1\%$1

with hyperparameters $\le 1\%$2, $\le 1\%$3, and $\le 1\%$4. The number of retained patch tokens is then

$\le 1\%$5

Higher confidence values below the early-exit threshold therefore induce more aggressive token dropping, while low-confidence examples retain more tokens (Wang et al., 3 Sep 2025).

TinyDrop then selects the top-$\le 1\%$6 tokens according to the saliency scores and concatenates them with the class token before feeding the shortened sequence into the unchanged target ViT. For absolute positional embeddings, the framework sub-indexes the original positional embedding tensor so that only the class token position and the positions of retained patch tokens remain. For relative position bias, the framework similarly extracts the corresponding submatrix. The target ViT therefore receives a shorter token sequence but otherwise retains its original architecture and weights. Because attention now operates on $\le 1\%$7 tokens instead of $\le 1\%$8, the attention cost is reduced roughly by $\le 1\%$9 for the attention components (Wang et al., 3 Sep 2025).

The method is explicitly training-free. There is no additional loss function, no fine-tuning, and no reparameterization of the target ViT. TinyDrop is implemented entirely through pre-trained models and input-side sequence reduction.

3. Guidance model design, implementation, and architectural compatibility

The guidance model $224 \times 224$0 is a lightweight image classifier chosen from off-the-shelf architectures. The reported guidance models are EfficientFormerV2$224 \times 224$1, with about $224 \times 224$2 GFLOPs at ImageNet resolution, and EfficientNetV2$224 \times 224$3, with about $224 \times 224$4 GFLOPs. Both are much smaller than ViT-B/16 or ViT-L/16 and provide feature maps suitable for Grad-CAM. The paper states that their small size makes the overhead of the guidance forward pass plus Grad-CAM minor relative to the saved FLOPs in large ViTs (Wang et al., 3 Sep 2025).

The framework is plug-and-play in the sense that the target backbone is untouched. There are no additional heads, no modified attention blocks, and no architectural changes to the ViT. In practice, the implementation is a wrapper that invokes the guidance model, computes Grad-CAM on its last block, resizes the saliency map to the ViT token grid, applies the confidence-to-drop mapping, subsamples the patch embeddings and positional encodings, and then forwards the reduced token sequence into the existing ViT. The paper states that this can be implemented at the API level without changing model definitions (Wang et al., 3 Sep 2025).

TinyDrop performs token dropping once, before the first transformer block. All subsequent layers process the shortened sequence length $224 \times 224$5. This single-stage schedule is described as simpler and more robust than multi-stage pruning schedules and as a source of substantial cumulative FLOPs reduction. The framework is evaluated on DeiT$224 \times 224$6, DeiT$224 \times 224$7, ViT$224 \times 224$8, ViT$224 \times 224$9, BEiTv2$16 \times 16$0, and DeiT3$16 \times 16$1. The stated assumptions are a standard patch embedding plus class token input format, positional embeddings or relative position bias that can be sub-indexed according to retained tokens, and a fixed patch grid so that the guidance saliency map can be resampled to the same spatial layout (Wang et al., 3 Sep 2025).

The reported experiments focus on ImageNet-1K classification. The authors note future extensions to multi-modal transformers and dense prediction tasks such as detection and segmentation, but those settings are not implemented in the current work.

4. Reported efficiency and accuracy

The principal reported result is that TinyDrop reduces FLOPs by up to $16 \times 16$2 for ViTs with minimal accuracy degradation, with more detailed operating points for large backbones reaching approximately $16 \times 16$3–$16 \times 16$4 reduction while preserving accuracy within small margins (Wang et al., 3 Sep 2025). The paper emphasizes that the FLOPs values include both the guidance model’s forward pass and the Grad-CAM backward pass.

Target ViT	Baseline	TinyDrop operating points
ViT$16 \times 16$5	61.6 GFLOPs, 85.8% top-1	8.0 GFLOPs (−87.0%) with $16 \times 16$6 drop; 11.1 GFLOPs (−82.0%) with $16 \times 16$7 drop; 14.0 GFLOPs (−77.3%) with $16 \times 16$8 drop
BEiTv2$16 \times 16$9	44.8 GFLOPs, 88.4% top-1	9.7 GFLOPs (−78.2%) with $T = 14 \times 14 = 196$0 drop; 12.8 GFLOPs (−71.3%) with $T = 14 \times 14 = 196$1 drop; 16.5 GFLOPs (−63.2%) with $T = 14 \times 14 = 196$2 drop
DeiT3$T = 14 \times 14 = 196$3	61.6 GFLOPs, 87.0% top-1	10.9 GFLOPs (−82.3%) with $T = 14 \times 14 = 196$4 drop; 16.0 GFLOPs (−74.0%) with $T = 14 \times 14 = 196$5 drop; 28.3 GFLOPs (−54.1%) with $T = 14 \times 14 = 196$6 drop

For smaller ViTs, the reported reductions remain large. DeiT$T = 14 \times 14 = 196$7 has a baseline of 17.6 GFLOPs and 81.8% top-1, while TinyDrop reports 2.9 GFLOPs (−83.5%) at 81.6% accuracy. ViT$T = 14 \times 14 = 196$8 has a baseline of 17.6 GFLOPs and 84.6% top-1, while TinyDrop reports 2.9 GFLOPs (−83.5%) at 83.0% accuracy. An explicit example given for ViT$T = 14 \times 14 = 196$9 reports 85.2% top-1 at 9.6 GFLOPs, compared with the baseline 85.8% at 61.6 GFLOPs, corresponding to a −84.4% FLOPs reduction (Wang et al., 3 Sep 2025).

The trade-off curves described in the paper indicate the expected monotonic relation between more aggressive dropping and lower FLOPs, with eventual accuracy degradation as $\mathcal{O}(T^2)$0 increases. The paper states that TinyDrop maintains accuracy well up to aggressive FLOPs reductions of roughly $\mathcal{O}(T^2)$1. Because the paper focuses on FLOPs rather than explicit wall-clock measurements, any inference about runtime acceleration should be treated as indirect rather than directly benchmarked.

5. Comparative position among token reduction and dynamic inference methods

On ImageNet, TinyDrop is compared with token reduction methods and early-exit baselines. For DeiT$\mathcal{O}(T^2)$2, the reported baseline is 79.8% at 4.6 GFLOPs. ToMe is reported at 79.4% and 2.7 GFLOPs, Zero-TP at 79.1% and 2.5 GFLOPs, PaPr at 79.2% and 3.0 GFLOPs, and TinyDrop at 79.6% and 1.4 GFLOPs. For DeiT$\mathcal{O}(T^2)$3, the baseline is 81.8% and 17.6 GFLOPs; DiffRate reports 81.5% and 11.5 GFLOPs, ToMe 81.4% and 11.5 GFLOPs, Zero-TP 81.0% and 13.6 GFLOPs, while TinyDrop reports 81.6% and 2.9 GFLOPs. For ViT$\mathcal{O}(T^2)$4, the baseline is 84.6% and 17.6 GFLOPs; ToMe reports 80.4% and 8.8 GFLOPs, ToFu 80.7% and 8.8 GFLOPs, PaPr 82.1% and 9.3 GFLOPs, while TinyDrop reports 83.0% and 2.9 GFLOPs. For ViT$\mathcal{O}(T^2)$5, the baseline is 85.8% and 61.6 GFLOPs; ToMe reports 83.5% and 31.0 GFLOPs, ToFu 83.9% and 31.0 GFLOPs, PaPr 83.9% and 31.1 GFLOPs, while TinyDrop reports 85.2% and 9.6 GFLOPs (Wang et al., 3 Sep 2025).

The paper characterizes these comparisons as showing larger FLOPs reductions for TinyDrop, typically around $\mathcal{O}(T^2)$6–$\mathcal{O}(T^2)$7, whereas prior methods in the table reach roughly $\mathcal{O}(T^2)$8–$\mathcal{O}(T^2)$9. It also states that accuracy remains competitive or better, especially at large compression ratios, while preserving a training-free formulation. The key methodological distinction is that TinyDrop uses an independent guidance model that supplies both saliency and confidence, whereas methods such as ToMe and Zero-TP infer token importance from internal token similarity or attention heuristics, and PaPr uses a small ConvNet but retains a fixed keep rate (Wang et al., 3 Sep 2025).

The work also compares TinyDrop with classical early-exit strategies such as CF-ViT, dynamic perceiver, and MSNet in accuracy–FLOPs plots. The paper states that TinyDrop points lie above others in these plots, meaning that for the same FLOPs TinyDrop attains higher accuracy, and for the same accuracy it requires fewer FLOPs. The claimed reason is the combination of a full-depth ViT backbone with input-dependent token dropping guided by a trained tiny model, rather than truncating the backbone through internal exits (Wang et al., 3 Sep 2025).

6. Ablations, limitations, and nomenclature

Ablation results isolate the effects of the confidence-to-drop mapping and the two constituent mechanisms of the framework. On BEiTv2, the curvature hyperparameter $\mathcal{M}_g$0 controls the accuracy–efficiency frontier. Small $\mathcal{M}_g$1 yields more aggressive dropping for moderate confidence, larger FLOPs savings, and a higher risk of discarding informative tokens. Large $\mathcal{M}_g$2 makes dropping more conservative, slightly improves robustness and accuracy, and reduces FLOPs savings. The paper reports that a mid-range value $\mathcal{M}_g$3 is used in the main experiments and gives the best trade-off (Wang et al., 3 Sep 2025).

The paper also reports ablations removing early exit or removing token dropping. For DeiT$\mathcal{M}_g$4, full TinyDrop gives 79.6% at 1.4 GFLOPs; without early exit it gives 79.1% at 3.0 GFLOPs; without token dropping it gives 79.6% at 1.8 GFLOPs. For DeiT$\mathcal{M}_g$5, full TinyDrop gives 81.6% at 2.9 GFLOPs; without early exit it gives 81.6% at 12.5 GFLOPs; without token dropping it gives 81.6% at 4.0 GFLOPs. For ViT$\mathcal{M}_g$6, full TinyDrop gives 83.0% at 2.9 GFLOPs; without early exit it gives 82.2% at 8.1 GFLOPs; without token dropping it gives 83.0% at 3.2 GFLOPs. For ViT$\mathcal{M}_g$7, full TinyDrop gives 85.2% at 9.6 GFLOPs; without early exit it gives 84.5% at 29.4 GFLOPs; without token dropping it gives 85.2% at 11.6 GFLOPs. These results are summarized in the paper as showing that token dropping alone already gives large savings, early exit alone also gives significant savings, and combining both yields the strongest overall efficiency (Wang et al., 3 Sep 2025).

Qualitative visualizations described in the paper show that tokens corresponding to foreground objects and salient regions are kept, whereas background or redundant regions are dropped. As $\mathcal{M}_g$8 varies, the density of retained tokens changes across the image. This suggests that the framework’s Grad-CAM-derived saliency is aligned with semantically important image regions, although the visualization evidence is qualitative.

The paper identifies several limitations. The effectiveness depends on the quality and calibration of the guidance model $\mathcal{M}_g$9; poor guidance can yield misleading saliency maps or miscalibrated confidence. The overhead of the guidance computation plus Grad-CAM backward pass is non-zero and may be less negligible for very small target models. The hyperparameters $\mathcal{M}_t$0, $\mathcal{M}_t$1, and $\mathcal{M}_t$2 govern the efficiency–accuracy trade-off and may require tuning for a given backbone or deployment setting. Experiments are limited to ImageNet classification, with no current implementation for detection, segmentation, or multi-modal tasks (Wang et al., 3 Sep 2025).

A nomenclature issue is that the string “TinyDrop” is not unique across arXiv-adjacent usage. In fluid dynamics, a tiny drop is discussed as a model system for ultra-thin air-film rupture and micro-bubble generation in “Bubble Chandeliers” (Thoroddsen et al., 2012), and a “TinyDrop” device is used descriptively in an overview of the slingshot droplet generator for producing small uniform droplets in air (Surabhi et al., 2019). In current machine-learning usage, however, TinyDrop denotes the specific ViT efficiency framework introduced in (Wang et al., 3 Sep 2025).