Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning

Abstract: Visual token pruning is a crucial strategy for accelerating VLMs by compressing redundant image patches, yet existing methods often fail to preserve critical cues under dense instructions and fine-grained queries. In this paper, we investigate this failure and identify two underlying bottlenecks: the widespread dispersion of textual noise that corrupts dense cross-modal scoring, and the feature fragmentation inherent to standard token selection. To address these issues, we propose Entropy-Aware Dense Pruning (EADP), a framework that reformulates pruning as a structured compression problem. EADP first leverages statistical entropy to quantify and filter out textual noise, yielding a robust, fine-grained instruction relevance score. Subsequently, instead of naive Top-K selection, EADP casts token selection as a submodular maximization problem with a spatial prior, explicitly ensuring a holistic and non-redundant visual representation. Extensive experiments demonstrate that EADP improves the accuracy-efficiency trade-off of VLMs, robustly preserving fine-grained visual cues under strict token budgets while achieving SoTA performance on challenging multimodal benchmarks.

• The paper introduces EADP, an entropy-aware framework that fuses dense guidance with global cues to denoise textual interference in visual tokenization.
• It employs spatial smoothing and submodular maximization to select coherent, non-redundant tokens, ensuring holistic visual coverage.
• Empirical results show up to 2.5 point accuracy gains and significant efficiency improvements across diverse Vision-Language Models and benchmarks.

Entropy-Aware Dense Visual Token Pruning for Multimodal Models

Motivation and Limitations of Existing Visual Token Pruning

Modern Vision-LLMs (VLMs) employ dense visual tokenization to capture fine spatial details, introducing significant computational challenges due to quadratic self-attention complexity. Compression by visual token pruning is essential for practical deployment, but existing strategies degrade under demanding conditions that involve dense instructions and fine-grained visual cues.

Standard pipelines use global textual guidance—typically the CLIP EOS token—to compute cross-modal token importance. This global representation, however, spreads attention over background regions and fails to consistently capture critical local cues, particularly for nuanced prompts and negative queries.

Figure 1: (a) Global guidance over-attends to background; (b) textual noise induces dispersed attention; (c) naive Top-$K$ selection fragments features and yields redundancy.

A further bottleneck is apparent when employing dense token guidance. Although matching each text token to every visual token seems more informative, the aggregation of functional words and punctuation—lacking semantic grounding—induces near-uniform, high-entropy responses over the spatial grid. Summing these obscures the signal of entity tokens, drowning crucial visual activations in textual noise.

At the selection stage, the prevalent Top-$K$ heuristic compounds these deficiencies by over-concentrating tokens in a few salient regions, leading to severe feature fragmentation (e.g., covering only a part of an object) and selection redundancy (multiple adjacent tokens capturing the same semantic entity), while neglecting semantic completeness.

Entropy-Aware Dense Pruning (EADP): Framework

To address these intertwined failures, the EADP framework reconceptualizes visual token pruning as a structured, noise-robust compression problem. The approach comprises a dual-stage pipeline: (1) entropy-aware instruction relevance scoring, and (2) spatially coherent, submodular token selection.

Figure 2: EADP compresses $N$ visual tokens into $K$ by (Stage 1) denoising dense guidance via entropy filtering and fusing with global score, and (Stage 2) spatially refining relevance followed by submodular selection for maximal representativeness.

Stage 1: Entropy-Guided Denoising for Instruction Relevance

1. Dense Cross-Modal Scoring: Compute a similarity matrix between all CLIP text tokens (excluding EOS) and projected visual tokens.
2. Dispersion Measurement: For each text token, softmax-normalize spatial similarities and compute information entropy. High-entropy tokens exhibit spatially diffused, uninformative attention; low-entropy tokens localize on semantic entities.
3. Noise Filtering: Retain a fixed quantile of tokens with lowest entropy and aggregate their similarities to form a noise-filtered dense relevance score.
4. Score Fusion: Fuse the denoised dense score with the global EOS-based score, balancing local precision and macroscopic consistency.

Stage 2: Spatial Smoothing and Submodular Selection

1. Spatial Refinement: Reshape the fused 1D relevance vector into a 2D map, smooth with a Gaussian kernel, and apply exponential score polarization to amplify salient regions and suppress background.
2. Facility Location Submodular Maximization: Reformulate token selection as maximizing the sum over all image tokens of their instruction relevance weighted by proximity (cosine similarity) to the selected set. This objective guarantees monotonicity and submodularity, enabling a fast greedy algorithm with $(1-1/e)$ approximation. The process avoids redundancy (clustering in high-score peaks) and enforces holistic visual coverage under tight token budgets.

Numerical Results and Empirical Analysis

Extensive experiments across multiple VLMs and benchmarks—spanning LLaVA-1.5, LLaVA-NeXT, Qwen2.5/3-VL, and LLaVA-Video—provide strong evidence for EADP's efficacy.

Key findings:

• On LLaVA-1.5-7B, EADP achieves the highest average accuracy under all token retention regimes, outperforming DivPrune, CDPruner, and others by up to 2.5 points at extreme compression (32 tokens).
• Robustness persists in high-resolution contexts (LLaVA-NeXT, 2880 tokens per image), maintaining near upper-bound accuracy even after losing over 90% of the tokens.
• In text-rich and document scenarios (Qwen3-VL's DocVQA, InfoVQA), EADP preserves holistic semantic structure, outperforming baselines by large margins at low budgets.
• Video understanding (LLaVA-Video) demonstrates EADP's capacity to preserve temporal-semantic integrity, with minimal performance drop at $>80\%$ token reduction.

Efficiency analysis shows that EADP offers substantial speedups ($\sim2.5\times$ inference latency reduction, $\sim66\%$ lower FLOPs at moderate budgets), with only minor computational overhead from entropy and submodular optimization modules. The architectural choices (entropy threshold, fusion coefficients, smoothing kernel) are robust to hyperparameter variation.

Theoretical and Practical Implications

EADP's adoption of entropy as a statistical, parser-free method to filter textual noise yields a noise-robust, locally precise instruction relevance measure, circumventing the need for external NLP tools or handcrafted rules. Submodular maximization reframes token selection not as peak-hunting, but as a representativeness objective intimately aligned with the lossy nature of token compression in high-dimensional visual-linguistic media. This mathematical guarantee translates to consistent downstream performance and resilience under aggressive compression.

Practically, EADP enables the deployment of VLMs in settings where low latency and restricted compute are non-negotiable, e.g., real-time multimodal agents and on-device inference. The plug-and-play design allows integration with diverse architectures without retraining or structural modification.

Future Directions

EADP's principles—structured entropy-guided filtering and submodular optimization—can be generalized to other multimodal compression and selection tasks, such as language token filtering under long-context regimes or dynamic patch allocation for streaming media. Further, the facility location objective lends itself to more advanced selection strategies (e.g., adaptive diversity constraints, mixture models for multimodal coverage). Progress in efficient submodular optimization and joint visual-linguistic scoring may yield even better trade-offs.

Conclusion

Entropy-Aware Dense Pruning (EADP) systematically addresses the dual challenges of textual noise and feature fragmentation in visual token pruning for VLMs. By quantifying and filtering out dispersed functional tokens using information entropy and modeling selection as a submodular maximization problem with spatial priors, EADP yields noise-robust, holistic, and non-redundant visual token sets. The demonstrated improvements in accuracy-efficiency trade-offs and theoretical guarantees position EADP as a rigorous and practical paradigm for scalable multimodal model deployment.

Reference:

"Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning" (2607.02484).