RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT

Abstract: Vision Transformers (ViTs) achieve strong performance but suffer from high computational costs due to quadratic self-attention complexity. Although token reduction techniques such as pruning and merging mitigate this, they typically overlook how representations evolve across network depth. We propose RAPID, a depth-aware token reduction framework that adapts reduction strategies to the layer-wise characteristics of token representations. The primary methodological contribution is a bifurcated strategy: in shallow-to-middle layers, RAPID employs a redundancy-similarity aware pruning metric to eliminate over-represented local patterns. As features transition to global semantic concepts in deeper layers, the framework shifts to an importance-similarity aware merging mechanism. This stage leverages classification (CLS) token attention weights to protect semantically critical tokens while fusing less important but similar neighbors. Empirical validation on ImageNet-1K using ViT and DeiT architectures demonstrates that RAPID establishes a superior accuracy-compression Pareto frontier compared to plug-and-play baselines such as ToMe and ToFu. RAPID is particularly robust in aggressive compression regimes, achieving up to 4.29% higher accuracy than ToMe at extreme reduction rates. Our framework provides a training-free template for optimizing vision models by aligning reduction strategies with hierarchical feature evolution.

• The paper presents a depth-adaptive framework that employs redundancy-aware pruning in early layers and importance-driven token merging in deeper layers.
• It achieves superior accuracy-efficiency trade-offs on ImageNet compared to baselines like ToMe and ToFu without requiring retraining.
• The method’s plug-and-play design and ablation analysis validate layer-wise token reduction tailored to evolving semantic representations.

RAPID: Depth-Aware Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient Vision Transformers

Introduction and Motivation

Vision Transformers (ViTs) have exhibited remarkable performance on vision tasks due to their ability to model long-range dependencies. However, the quadratic complexity of their self-attention mechanisms imposes substantial computational and memory overhead, making them less suitable for resource-constrained applications. Prior work has addressed this limitation either by redesigning the backbone architecture or by reducing computational demand through token pruning and/or merging. Architectural approaches require retraining and are less flexible for newly emerging pretrained models. Token pruning/merging strategies often overlook the representational shift that occurs across network depth, leading to information loss or suboptimal fusion dynamics.

The RAPID framework addresses these limitations by introducing a layer-wise, depth-aware token reduction strategy that distinguishes between the roles of pruning and merging as token representations evolve. Specifically, in early layers, RAPID employs redundancy-similarity aware pruning, while in deeper layers, it switches to importance-similarity driven token merging, leveraging the semantic richness encoded by tokens at different depths. This bifurcated approach aligns the token reduction method with the evolving content and function of tokens in the ViT pipeline.

Methodology

The central insight behind RAPID is the empirical observation that token semantics in ViTs transition from encoding local, low-level visual patterns in early layers to representing global, high-level semantic concepts in deeper layers. Accordingly, a single uniform token reduction strategy results in information loss (if aggressive pruning is used late) or inadequate compression (if conservative merging is used early). RAPID adaptively applies two tailored algorithms in different network stages:

Redundancy-Similarity Aware Pruning (Shallow-to-Mid Layers)

In the shallow-to-mid layers, where patch tokens primarily represent redundant local patterns, RAPID ranks tokens using a redundancy-similarity metric. Tokens are partitioned into two groups, and pairwise cosine similarity is computed. Tokens with high aggregate similarity margins—indicating strong redundancy relative to a threshold—are removed.

Figure 1: RAPID's architecture alternates between redundancy-similarity aware pruning in early layers and importance-similarity aware merging in later layers, reflecting the hierarchical evolution of token representations.

This mechanism preferentially prunes tokens whose removal least distorts the overall representation, thereby preserving diversity and minimizing accuracy degradation under aggressive reduction.

Importance-Similarity Aware Merging (Mid-to-Late Layers)

In deeper layers, token representations increasingly encapsulate semantic information. Here, RAPID protects tokens with high importance, as measured by their attention from the CLS token, and only merges less important, yet similar, tokens. An importance offset, controlled by a hyperparameter $\lambda$, adjusts the merge score to ensure semantically critical atoms are not inadvertently fused.

The merging operation proceeds by selecting the top token pairs (based on adjusted similarity scores penalized for importance) for fusion, ensuring that essential semantic content is retained and information from less important but similar tokens is consolidated efficiently.

Switching Strategy and Implementation Details

Ablation analysis demonstrates a marked performance advantage when the reduction strategy transitions from pruning to merging after the fourth layer, confirming that redundancy-based pruning is most suitable early, followed by importance-aware merging as global structure emerges. RAPID is implemented as a plug-and-play module, requiring no retraining, and is compatible with standard ViT backbones via simple API wrapping.

Experimental Results

Evaluation on ImageNet-1K using multiple ViT and DeiT variants validates the efficacy of RAPID. The framework is benchmarked under aggressive token reduction regimes and compared against strong baselines ToMe and ToFu.

Figure 2: Top plot—Compression rate vs Top-1 accuracy; Bottom plot—Peak GPU memory vs Top-1 accuracy. RAPID outperforms ToMe and ToFu at comparable compression levels, especially in high compression (low token retention) settings.

Key findings include:

• Superior Robustness under High Compression: RAPID achieves up to 4.29% higher accuracy than ToMe in extreme reduction regimes.
• Favorable Pareto Frontier: RAPID consistently maintains higher Top-1 accuracy for a given compression rate or memory footprint, demonstrating greater resilience to token thinning.
• Plug-and-Play Flexibility: Without requiring additional finetuning, RAPID matches or exceeds the performance of baseline methods that allow direct inference-time token reduction.
• Ablation Validates Methodological Decomposition: Switching from redundancy-pruning to importance-merging after the fourth layer yields the strongest results, highlighting the criticality of depth-aware strategy selection.

The results further reveal that for deeper ViT variants (e.g., ViT-L), RAPID is more robust to drastic token reductions than generic, non-adaptive approaches.

Implications and Future Directions

RAPID's decoupling of reduction strategies by layer depth introduces several practical and theoretical implications:

• Deployment: The training-free, modular design facilitates application to new and legacy pretrained ViT models, enabling easy integration into real-world inference workloads where resources are strictly limited.
• Theoretical Insight: The observed alignment between token compression strategies and hierarchical representation semantics in ViTs may inform future research into adaptive or learned layer-wise policies, potentially generalizing to multimodal transformers and hierarchical self-attention mechanisms.
• Broader Applicability: While demonstrated on image classification, the framework's conceptual architecture—layer-dependent preservation vs. fusion—may extend to tasks requiring structured prediction (e.g., segmentation, detection) and even to visual-LLMs.
• Automated Policy Search: Future avenues include automating the switching strategy, possibly through differentiable controllers or data-driven triggers, and extending the redundancy/importance score definitions to account for dynamic task-specific signals.

Conclusion

RAPID provides a principled, depth-adaptive token reduction framework for ViTs, achieving state-of-the-art accuracy-efficiency trade-offs without additional training. By explicitly tailoring redundancy-pruning and importance-merging strategies to hierarchical token semantics, RAPID minimizes information loss and enhances model robustness under constrained computational budgets. This work highlights the significance of aligning compression mechanisms with representation evolution in deep transformer architectures and opens avenues for principled modularization of token reduction across diverse transformer-based domains.