TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference

Abstract: Multimodal LLMs (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial computational overhead. Visual token pruning offers a natural solution, yet existing methods are imperfect: attention-based criteria tend to retain redundant tokens, while diversity-based criteria are often agnostic to user instructions. Even methods that combine multiple criteria still lack a principled formulation of the intrinsic objective of token pruning. In this paper, we revisit visual token pruning from a first-principles perspective and formulate it as constructing Token Optimal Preservation Sets. Through a top-down information-theoretic analysis, we identify three fundamental principles for effective token selection: Task Relevance, Information Coverage, and Semantic Diversity. Based on these principles, we propose TOPS, a training-free and model-agnostic pruning module that can be applied to various MLLMs. Extensive experiments on 7 MLLM backbones and 14 benchmarks demonstrate that TOPS outperforms prior methods under diverse pruning settings. Notably, on LLaVA-NeXT, TOPS removes 77.8% of visual tokens while preserving 100.0% and 100.6% performance on its 7B and 13B models, respectively, suggesting that pruning redundant visual tokens can sometimes mitigate hallucination and inspire future lightweight MLLM design.

• The paper introduces TOPS, which uses an information-theoretic objective to balance task relevance, information coverage, and semantic diversity in token pruning.
• It employs a two-stage, training-free, and model-agnostic pruning strategy—first post-vision encoding and then within transformer layers—to ensure robust MLLM inference.
• Experimental results demonstrate up to 77.8% token reduction with maintained or improved accuracy, alongside nearly 10× FLOPs reduction and lower GPU memory usage.

TOPS: First-Principles Visual Token Pruning for Efficient MLLM Inference

Motivation and Problem Formulation

Contemporary MLLMs are fundamentally over-parameterized with respect to visual token throughput: high-resolution visual input yields large token sequences, which introduces major inefficiency due to the quadratic scaling of self-attention. Existing methods for visual token pruning have been largely heuristic, relying on isolated criteria such as attention magnitude, token diversity, or information coverage. These approaches tend to either retain redundant tokens, neglect task-relevant evidence, or discard useful diversity. Critically, prior work lacks a principled objective for defining the information that must be preserved through pruning.

TOPS reframes visual token pruning from a first-principles, information-theoretic perspective. Rather than relying on independent scoring rules, TOPS formalizes pruning as constructing a Token Optimal Preservation Set: the smallest subset of visual tokens such that MLLM reasoning remains as faithful as possible to the unpruned model. This is cast as maximizing the mutual information $I(S; V, Q)$ between the retained set $S$, the full visual token set $V$, and the query $Q$. Decomposition of this quantity yields three orthogonal desiderata for the optimal subset: Task Relevance ($I(S; Q)$), Information Coverage ($I(S; V|Q)$), and Semantic Diversity (each token contributes non-redundant information).

Methodology: The TOPS Algorithm

TOPS introduces a two-stage, dynamic selection strategy that is both training-free and model-agnostic, applicable to any transformer-based MLLM.

• Stage I (Vision-Side Pruning): Applied post-vision-encoder, pre-LM, using CLS-to-patch attention to score relevance. This eliminates tokens that are globally redundant prior to entering the text-conditioned fusion layers.
• Stage II (In-LLM Layer Pruning): Applied at user-chosen transformer layers, leveraging text-guided attention signals for fine-grained, query-aware pruning. At each layer and each selection step, the candidate token maximizing a weighted sum of normalized task relevance, information coverage (based on token-to-token sim matrix), and semantic diversity (max distance from current preserved set) is greedily selected.

Let $\tilde{r}_i$, $\tilde{c}_i(U)$, and $d_i(S)$ denote normalized task relevance, information coverage, and semantic diversity for candidate $i$. The selection score is:

$S$0

where $S$1 are tunable weights. The process continues until the budget $S$2 is exhausted. Coverage and diversity metrics are efficiently maintained incrementally, introducing marginal overhead. Dynamic text-rater sets (subset of text tokens strongly coupled to vision) are used for query-aware attention collection.

Experimental Results

The authors conduct an extensive empirical study across 7 diverse MLLMs (LLaVA-1.5, LLaVA-NeXT, LLaVA-Video, InternVL3-8B, Qwen2.5-VL-7B, etc.) and 14 multimodal benchmarks (GQA, POPE, MME, AI2D, HallusionBench, MLVU, Video-MME, etc.), benchmarking against FastV [attention-based], DivPrune [diversity], SCOPE [coverage-based], and hybrid/token-merging methods.

Key Results

• Token Compression: On LLaVA-NeXT-7B, TOPS removes 77.8% of visual tokens while maintaining 100.0% relative accuracy; for LLaVA-NeXT-13B, accuracy is actually slightly increased (100.6%), implying that aggressive redundant token removal can regularize and mitigate model hallucination.
• Aggressive Pruning: Even under extreme compression (down to 5.6% tokens, i.e., 32 of 576), TOPS outperforms all baselines by $S$3 relative accuracy on vision-language benchmarks.
• Generalization: On Qwen2.5-VL-7B and InternVL3-8B, TOPS delivers $S$4 to $S$5 higher accuracy than the best alternative (CDPruner/DivPrune) at similar pruning ratios.
• Efficiency: TOPS achieves close to 10$S$6 FLOPs reduction, 3$S$7 lower prefill latency, and $S$82 GB lower GPU memory (on LLaVA-NeXT-7B @ POPE), without sacrificing performance.
• Ablations: Task relevance dominates at moderate compression, while coverage and diversity become increasingly important as budgets shrink. The optimal synergy is obtained when all three criteria are combined.

Robustness and Layerwise Analysis

A key design choice is multi-stage, progressive pruning. Cross-layer Jaccard analysis reveals that importance assignments are unstable across transformer layers, thus greedy token selection must be staged for effective redundancy elimination. Ablations confirm that mid-to-late transformer layers (e.g., {12,24} or {15,30}) yield the best efficiency–accuracy tradeoffs, as early pruning (before multimodal fusion matures) can lose critical evidence.

Practical and Theoretical Implications

From a theoretical standpoint, this framework positions token pruning as a subset selection problem with a grounded, information-theoretic objective, obviating reliance on task- or architecture-specific heuristics. The incorporation of diversity and coverage as submodular objectives (with provable diminishing marginal utility per additional token) justifies the greedy algorithm and ensures robust selection. Practically, TOPS enables stable MLLM deployment under severe resource constraints (e.g., mobile inference, edge devices), with potential for on-the-fly adaptation to input complexity, task instruction, or dynamic query demands.

Notably, the training-free nature of TOPS ensures model-agnosticism and compatibility with proprietary or black-box MLLM deployments, lowering the hardware and energy cost of state-of-the-art vision-language reasoning at scale.

Critical Perspective and Future Directions

While TOPS delivers consistent improvements, it introduces $S$9 selection overhead in the greedy loop—non-trivial for highly compressed sequences or many pruning locations. Extensions using approximate or parallel diversity/coverage estimators could further reduce runtime. The reliance on standard attention matrices for task relevance could face challenges with future transformer variants adopting alternative attention forms. Adaptive weighting of the three criteria—possibly in a learned or meta-optimized manner, rather than fixed—may boost performance for highly domain-specialized tasks.

Furthermore, systematic evaluation across dense and domain-specific modalities (e.g., medical imaging, remote sensing, ultralong temporal sequences) is required to fully characterize the universality of the three-principle framework.

Conclusion

TOPS provides a rigorous, information-theoretic foundation for visual token pruning in MLLMs, achieving architecture-agnostic, training-free, and highly efficient inference with negligible accuracy degradation—even under aggressive compression. By explicitly prioritizing task relevance, information coverage, and semantic diversity, TOPS shifts the paradigm from heuristic importance scoring to optimal subset construction. This method constitutes a reference approach for future research on efficient multimodal inference and can inspire principled design of lightweight, scalable MLLMs.