ID-Selection: Importance-Diversity Based Visual Token Selection for Efficient LVLM Inference

Abstract: Recent advances have explored visual token pruning to accelerate the inference of large vision-LLMs (LVLMs). However, existing methods often struggle to balance token importance and diversity: importance-based methods tend to retain redundant tokens, whereas diversity-based methods may overlook informative ones. This trade-off becomes especially problematic under high reduction ratios, where preserving only a small subset of visual tokens is critical. To address this issue, we propose ID-Selection, a simple yet effective token selection strategy for efficient LVLM inference. The key idea is to couple importance estimation with diversity-aware iterative selection: each token is first assigned an importance score, after which high-scoring tokens are selected one by one while the scores of similar tokens are progressively suppressed. In this way, ID-Selection preserves informative tokens while reducing redundancy in a unified selection process. Extensive experiments across 5 LVLM backbones and 16 main benchmarks demonstrate that ID-Selection consistently achieves superior performance and efficiency, especially under extreme pruning ratios. For example, on LLaVA-1.5-7B, ID-Selection prunes 97.2% of visual tokens, retaining only 16 tokens, while reducing inference FLOPs by over 97% and preserving 91.8% of the original performance, all without additional training.

• The paper introduces a training-free token selector that combines visual or instruction-based importance with iterative similarity suppression, retaining diverse, relevant tokens without expensive subset optimization.
• ID-Selection preserves 91.8% of full-model performance while pruning 97.2% of LLaVA-1.5-7B visual tokens, outperforming CDPruner and VisPruner under the same 16-token budget.
• Across five LVLMs and 16 benchmarks, the method delivers up to a 9.9× FLOPs reduction, 2.5× speed-up, and over 89% KV-cache reduction, with especially strong gains on text-focused tasks.

Motivation and problem statement

Large vision-LLMs (LVLMs) convert visual inputs into long token sequences that dominate inference cost, particularly for high-resolution images and multi-frame video. Qwen2.5-VL can process up to 16,384 visual tokens per input, far exceeding typical text-only sequence lengths. Visual token pruning has emerged as the dominant mitigation strategy, but existing approaches exhibit a systematic trade-off. Importance-based methods such as FastV (cross-modal attention) and FasterVLM ([CLS] attention) identify informative tokens yet favor tokens clustered in locally dense feature-space regions, retaining substantial redundancy. Diversity-based methods such as DART and DivPrune produce dispersed subsets but may discard instruction-relevant or visually salient tokens. Hybrid methods address this incompletely: VisPruner merges independently selected importance and diversity subsets in a disjoint manner, while CDPruner casts selection as conditional determinantal point process (DPP) optimization at higher computational cost.

The paper's central claim is that importance and diversity must be balanced jointly within a single selection process, and that this balance matters most under extreme pruning ratios where only a handful of tokens survive.

Method

ID-Selection is a training-free, plug-and-play token selection procedure with two components.

Importance estimation. Each visual token receives an initial importance score. For LLaVA-family models, three estimators are evaluated: (i) cross-modal attention from the last instruction token in an early LLM layer, following FastV; (ii) [CLS] attention from the visual encoder as a visual saliency measure; and (iii) a unified score combining both cues — cosine similarity between each visual embedding and an instruction feature extracted from the paired text encoder (CLIP/SigLIP), min-max normalized and multiplied element-wise with the [CLS] attention score. The unified score highlights regions that are simultaneously salient and question-relevant. For architectures without a paired text encoder, namely Qwen2.5-VL and InternVL2.5, the method falls back to cross-modal attention alone.

Diversity-aware iterative selection. Starting from an empty set $\mathbf{R}$, the algorithm repeatedly selects the highest-scoring remaining token, then suppresses the scores of all remaining tokens proportionally to their similarity to the selected one:

$$w_{ij}=\exp\left(-\gamma \cdot d(i,j)^2\right), \qquad S_j \leftarrow S_j - w_{ij}\cdot S_i,$$

where $d(i,j)$ is cosine distance and $\gamma$ is fixed at 20 across all experiments. Similar tokens are thus deprioritized softly rather than hard-removed, allowing informative-but-distinct tokens to survive. Each iteration costs $O(N)$, giving overall complexity $O(NT)$ with no matrix inversion or global subset optimization — a deliberate contrast with DPP-based selection.

Empirical results

The evaluation spans 5 LVLM backbones (LLaVA-1.5-7B, LLaVA-Next-7B, Video-LLaVA, Qwen2.5-VL-7B, InternVL2.5-8B) and 16 benchmarks covering general image understanding, text-rich document understanding, and video QA, against 9 baselines including FastV, FasterVLM, SparseVLM, PruMerge, VisionZip, DART, DivPrune, VisPruner, and CDPruner.

The headline result is on LLaVA-1.5-7B with only 16 of 576 tokens retained (97.2% pruning): ID-Selection with the unified importance score preserves 91.8% of full-model performance while cutting FLOPs by over 97%, without any retraining. At this extreme budget it outperforms CDPruner by roughly 0.9 points on average and VisPruner by 11 points. A representative comparison under high compression:

Method	Retained tokens	Avg. performance (% of full model)
FasterVLM	16 / 576	82.7
DivPrune	16 / 576	86.2
VisPruner	16 / 576	80.8
CDPruner	16 / 576	90.9
ID-Selection (unified)	16 / 576	91.8

On LLaVA-Next-7B, ID-Selection with [CLS] attention retains 95.0% of original performance at a 94.4% reduction ratio, exceeding CDPruner and VisPruner. On Video-LLaVA, reducing 2048 tokens to 128 still outperforms all baselines. On Qwen2.5-VL-7B and InternVL2.5-8B — architectures lacking a paired text encoder — the cross-modal attention variant retains 92.6% and 93.5% respectively at an 88.9% reduction ratio, confirming architectural generality.

The strongest relative gains appear on text-focused benchmarks, where dense textual cues are hardest to preserve. Under 88.9% reduction on Qwen2.5-VL, FastV retains only 53.4% of original performance versus 70.5% for ID-Selection; on InternVL2.5 the gap is 44.2% versus 60.5%. This indicates the diversity-aware suppression is particularly effective when fine-grained content must be preserved within very few tokens.

Efficiency measurements on LLaVA-Next-7B (POPE benchmark, single A800 GPU) show a 9.9× FLOPs reduction, 2.5× wall-clock speed-up, and over 89% KV cache reduction at an 88.9% pruning ratio. Notably, ID-Selection runs faster than CDPruner (910–916 s vs. 1010 s total evaluation time) at identical FLOPs and cache footprint, supporting the claim that iterative score suppression carries lower practical overhead than DPP-based subset optimization. Adding the text encoder for the unified score incurs negligible extra cost.

Ablations

Three ablations isolate the contributions. First, comparing ID-Selection against top-k selection under identical importance estimators (FastV and FasterVLM as counterparts) shows consistent gains, demonstrating that the improvement stems from the iterative suppression mechanism itself rather than the importance metric. Second, the choice of importance cue is task-dependent: [CLS] saliency performs better on TextVQA, instruction relevance on GQA, and the unified score dominates across tasks, indicating neither modality-specific cue suffices alone. Third, performance improves monotonically with $\gamma$ before saturating, which the authors interpret as evidence that redundancy concentrates among highly similar tokens.

Limitations and open questions

Several constraints should be noted. The unified importance score requires a text encoder paired with the visual encoder, restricting its applicability to architectures such as Qwen2.5-VL and InternVL2.5, where the paper defaults to a weaker single-cue estimator. The suppression hyperparameter $\gamma$ is fixed empirically at 20; whether it transfers robustly to unseen models or budgets is not analyzed. The suppression update subtracts a weighted fraction of the selected token's score, so behavior depends on the scale properties of the chosen importance metric — a potential fragility if scores are poorly calibrated. Finally, the method prunes tokens entirely rather than merging them, leaving open whether soft suppression could be combined with token aggregation to recover information discarded at extreme ratios, and how the approach interacts with adaptive-resolution pipelines that vary token counts per input.

Conclusion

ID-Selection couples importance estimation with diversity-aware iterative score suppression in a single training-free selection loop, achieving state-of-the-art accuracy–efficiency trade-offs across 5 LVLMs and 16 benchmarks, with its clearest advantage under extreme compression (91.8% performance retention at 97.2% token pruning on LLaVA-1.5-7B) and on text-dense benchmarks. Its low per-step overhead makes it more practical than DPP-based alternatives, though reliance on paired text encoders for the unified score and a fixed suppression parameter remain open design questions.