Token Pruning in Transformer Models

• Token pruning is a method that reduces redundancy by discarding low-utility tokens to counteract the quadratic complexity of self-attention.
• It leverages importance-based, sensitivity, and diversity-aware metrics to select a critical token subset for efficient inference.
• Empirical results show token pruning can achieve significant speed-ups (up to 12.1×) and FLOPs reduction with minimal accuracy degradation.

Token pruning is a class of methodologies in Transformer-based models—spanning vision, language, audio, and multimodal domains—that systematically reduces the number of input or intermediate tokens to achieve significant computational acceleration with minimal task performance degradation. By identifying and selectively discarding lower-utility tokens, pruning methods exploit the quadratic cost scaling of self-attention and enable efficient inference and/or memory footprint in high-resolution and long-context scenarios.

1. Theoretical Motivation and Problem Scope

The central challenge addressed by token pruning is the quadratic computational and memory cost of self-attention on large token sets. For an input of $N$ tokens, each Transformer layer incurs $O(N^2 d)$ time and $O(N^2)$ KV-cache memory. In dense tasks—such as high-resolution visual understanding, video-language modeling, or document retrieval—this constraint is amplified as $N$ may reach thousands. The objective is to retain only a critical subset $X' \subset X$, $|X'| \ll |X|$, such that the model’s predictive power $I(X';Y) \approx I(X;Y)$ is maintained up to a small tolerance, while greatly reducing computation and storage (Wen et al., 17 Feb 2025, Liao et al., 2024).

Approaches to token pruning can be categorized as (i) importance-based (scoring tokens via attention, gradients, or statistical measures), (ii) diversity-promoting (maximizing spatial or feature coverage), and (iii) hybrid/structural (balancing importance and redundancy while enforcing spatial or semantic diversity) (Zhang et al., 22 Dec 2025).

2. Methodological Principles and Scoring Criteria

2.1 Attention- and Cross-Attention-Based Measures

Many early approaches leverage self- or cross-attention scores as proxies for token “importance.” For multimodal models, cross-attention between text and vision streams provides a direct signal for visual token relevance to a linguistic query. One example is CATP (Liao et al., 2024), which aggregates voting scores across heads, layers, and optionally image-token importance to rank and prune decoder tokens. The per-token importance $I_i$ is computed as

$$I_i = \sum_{l=1}^L \sum_{h=1}^H \sum_{j=1}^{L_1} w_j v_{j\rightarrow i}^{(l,h)}$$

where $v_{j\rightarrow i}$ uses a descending ranking of attention weights.

Another robust direction exploits the model’s own cross-attention to define relevance, as in PruneVid for video QA (Huang et al., 2024) and window-averaged cross-modal attention for token selection in vision-LLMs (Wen et al., 17 Feb 2025).

2.2 Sensitivity and Gradient-Free Metrics

Recent work identifies limitations in raw attention scores, such as instability and positional bias, and proposes direct measures of token sensitivity. ZOO-Prune (Kim et al., 29 Sep 2025) estimates a token’s downstream influence by applying random perturbations at the (typically shallow) projection layer and using finite differences to approximate the effect on the model’s output—requiring only forward evaluations and no backpropagation. The token score is

$O(N^2 d)$0

where $O(N^2 d)$1 is the projection, $O(N^2 d)$2 is a random direction, and $O(N^2 d)$3 is the step size.

Relatedly, TransPrune (Li et al., 28 Jul 2025) introduces token transition variation (TTV), combining the magnitude and directional change of a token embedding across transformer layers to directly capture representational novelty:

$O(N^2 d)$4

2.3 Diversity, Structural Constraints, and Hybrid Schemes

Structural blind spots—such as spatially clustered selections or overemphasis on duplicated context—are remedied by approaches enforcing spatial or semantic diversity. D²Pruner (Zhang et al., 22 Dec 2025) and Balanced Token Pruning (BTP) (Li et al., 28 May 2025) combine debiased importance measures with structural selection via Maximal Independent Set (MIS) algorithms or staged calibration. D²Pruner computes importance debiased by a positional prior:

$O(N^2 d)$5

where $O(N^2 d)$6 is the observed attention, $O(N^2 d)$7 is a location-wise prior, and then applies graph-based selection to maximize coverage and distinctiveness.

Other work, such as IWP (Lee et al., 1 Apr 2026), reframes pruning in the dual of attention, selecting token subsets whose induced sum of rank-1 outer products best approximates the original linear layer, using metrics for both magnitude and duplication.

HiPrune (Liu et al., 1 Aug 2025) leverages hierarchical attention analysis in vision-LLMs, identifying that object-centric tokens dominate mid-layer attention while later layers reflect global context, and selects a mixed pool of anchor (object), buffer (spatial neighbor), and register (global) tokens.

3. Algorithmic Pipelines and Calibration

Most token pruning methods follow a pipeline:

• Compute token importance/diversity statistics—via attention, sensitivity, TTV, or precomputed priors—at one or more stages in the model.
• Optionally, spatial or semantic clustering merges or constrains candidate tokens to ensure uniform coverage or reduce redundancy.
• Sort tokens according to calibrated objective(s), such as maximizing a convex combination of importance and pairwise distance (Wen et al., 17 Feb 2025), then select the top-$O(N^2 d)$8 or spatially-balanced set under compute budget constraints.
• Prune the remainder; optionally, perform “soft” pruning by merging or recovering information (as in DaTo (Zhang et al., 2024) for diffusion models or SViT (Liu et al., 2023) for detection/segmentation).

Calibration frequently involves a small, fixed set of images/prompts, used to identify pruning layers (semantic-shift points) and stagewise retention fractions that align with representational dynamics (Li et al., 28 May 2025).

Table: Scoring Methodologies and Target Models

Method	Core Criterion(s)	Applicable Models/Tasks
CATP	Cross-attention voting	VQA, BLIP-2
ZOO-Prune	Zeroth-order sensitivity + diversity	CLIP-ViT VLMs
D²Pruner	Debiased importance + MIS-diversity	LLaVA, Qwen2.5-VL, InternVL
HiPrune	Hierarchical attention anchors/buffers	ViT-based VLMs
Balanced Token Pruning	Local-global joint loss (attention + diversity)	LLaVA, Qwen2.5-VL
FitPrune	Calibration-based attention distribution fit	MLLMs (e.g., LLaVA-NEXT)
PruneVid	Temporal/Spatial token merge + question-aware pruning	Video LLMs

4. Empirical Results and Trade-offs

Across tasks and architectures, token pruning methods consistently report significant reductions in compute with small accuracy losses. CATP (Liao et al., 2024) achieves up to 12.1× accuracy improvement over prior baselines at fixed prune rates in VQA. ZOO-Prune prunes up to 94.4% of tokens while maintaining ≥95% of task accuracy and up to 2.3× speed-up in end-to-end inference over unpruned models (Kim et al., 29 Sep 2025). D²Pruner preserves 99.2% of performance in understanding tasks at a 74.2% FLOPs reduction and supports extreme 90% pruning rates in localization without catastrophic failure (Zhang et al., 22 Dec 2025). PruneVid yields >80% token pruning and sometimes increases VideoQA accuracy (Huang et al., 2024).

Aggressiveness of pruning must be calibrated; typical regimes retain 22–33% of tokens for minimal loss, while pruning below 10% may degrade fine-grained metrics or spatial localization (Liu et al., 1 Aug 2025, Zhang et al., 22 Dec 2025). Techniques such as buffering, spatial-diversity enforced selection, or attention debiasing mitigate these edge effects.

5. Architectural Integration and Task Specialization

Token pruning adapts to architecture and task requirements:

• Visual Transformers for classification, detection, segmentation: importance scores from self-attention or lightweight gating networks (e.g., SViT (Liu et al., 2023)); care must be taken not to discard tokens irrevocably in dense tasks—preservation and reactivation are critical.
• Vision-LLMs: both raw- and cross-attention-based importance scoring; hybrid debiased and diversity-aware methods; plug-and-play nature supports drop-in acceleration.
• Audio Transformers: MAC reduction strategies leverage attention-based TopK selection; low-energy tokens are critical for general audio classification (Lee et al., 2 Apr 2025).
• Video-LLMs: spatio-temporal merging and question-aware relevance are especially effective in reducing redundancy (Huang et al., 2024).
• Diffusion Transformers: pruning of reference/context tokens leverages temporal update strategies and influence metrics for high-fidelity image-to-image editing (Lin et al., 2 Feb 2026).

Specializations include background-aware modules (e.g., BAViT (Sah et al., 2024)), MIS-based selection for spatial consistency, and progressive pruning for in-context or streaming scenarios.

6. Limitations, Evaluation Protocols, and Future Directions

Recent meta-analyses identify common pitfalls in token pruning research. Attention-based methods suffer positional/coverage bias (e.g., clumping on later tokens), redundancy-unaware loss, and may underperform windowed random pruning (Wen et al., 17 Feb 2025). Diversity-only schemes can miss prompt-conditioned relevance. Evaluation protocols based solely on FLOPs often misestimate realized speedup, especially if kernel/FlashAttention incompatibilities or late-layer pruning predominate.

For best practice, real latency, KV-cache savings, and throughput should be reported in addition to theoretical FLOPs. Future methods are advised to combine spatial uniformity, calibrated language guidance, explicit redundancy suppression, and hardware-awareness. Integration with token merging at model pre-training and fine-tuning stages may further enhance real-world efficiency (Wen et al., 17 Feb 2025).

Extensions to video, 3D, audio, and multi-modal streams demand adaption of both scoring criteria and selection mechanics, particularly as longer-context and edge deployment requirements intensify. Adaptive and task-conditioned methods, as well as learned pruning schedules, remain active research frontiers.

7. Conclusion

Token pruning provides a rigorous and practical acceleration strategy for Transformer-based models in vision, language, audio, and multimodal tasks. By leveraging importance and redundancy-aware metrics, hybrid diversity-promoting selections, and calibration with minimal data, modern methods routinely achieve 2–9× computational reductions with 1–3% accuracy cost under standard operating regimes. Recent work emphasizes the need to overcome bias, enforce spatial/semantic coverage, and harmonize evaluation metrics with practical deployment targets. Token pruning forms a crucial component in the scalable deployment of large-scale Transformer architectures across modalities and workloads.