AgilePruner: Adaptive Visual Token Pruning

• AgilePruner is an adaptive visual token pruning framework that balances attention concentration and feature diversity to reduce computational cost while limiting accuracy loss and hallucinations.
• It leverages analytical measures like attention entropy and effective rank to assess image complexity and guide a dynamic, threshold-based token selection process.
• Empirical results show AgilePruner achieving near-optimal performance across benchmarks, highlighting improved trade-offs between recall, grounding fidelity, and hallucination reduction.

AgilePruner is an adaptive visual token pruning framework for large vision-LLMs (LVLMs) that studies, and then exploits, the complementary behavior of attention-based and diversity-based pruning. Rather than treating those families as interchangeable compression heuristics, it analyzes what they preserve, how they fail, and how their utility varies with image complexity. The resulting method is a minimal threshold-based pre-pruning procedure that adjusts the balance between attention concentration and feature diversity on a per-image basis, with the stated objective of reducing visual token cost without sacrificing too much accuracy or increasing hallucinations (Baek et al., 1 Mar 2026).

1. Problem setting and design motivation

AgilePruner addresses the computational burden created by the large visual token sequences used in LVLMs. Because transformer attention scales quadratically with sequence length, passing hundreds or thousands of visual tokens into the LLM incurs substantial FLOPs, latency, and memory cost. Visual token pruning therefore aims to remove redundant tokens before they reach the LLM, but pruning is intrinsically risky: if the retained subset omits the wrong evidence, downstream answers may degrade or the model may hallucinate objects absent from the image (Baek et al., 1 Mar 2026).

The framework is situated against two dominant lines of prior work. Attention-based methods prioritize tokens receiving strong model attention, typically favoring concentrated evidence and conservative grounding. Diversity-based methods attempt to retain a broader spread of features, aiming to preserve scene coverage. AgilePruner’s central premise is that the field has often treated these approaches as competing heuristics without sufficiently characterizing their actual behavior. The paper instead argues that they preserve different kinds of information, fail in different regimes, and should be combined adaptively rather than by fixed mixing ratios (Baek et al., 1 Mar 2026).

A key conceptual distinction is that AgilePruner is a pre-pruning method. In the reported setup, pruning is performed before the visual tokens enter the LLM, and the paper states that this yields stronger efficiency gains than pruning inside the LLM. This places the method within the broader effort to improve LVLM inference efficiency while preserving grounding fidelity and downstream reasoning quality (Baek et al., 1 Mar 2026).

2. Analytical framework: attention entropy and effective rank

The empirical core of AgilePruner is built on two diagnostic quantities. The first is attention score entropy, used to quantify how concentrated the class-token attention is over visual tokens. If $p$ denotes the normalized class-token attention distribution, the entropy is

$H(p) = - \sum_i p_i \log p_i.$

Lower entropy indicates that attention is concentrated on a few salient regions, whereas higher entropy indicates more diffuse attention (Baek et al., 1 Mar 2026).

The second quantity is effective rank, or $erank$, which measures the effective dimensionality of the retained token embeddings. For a token embedding matrix $A \in \mathbb{R}^{N \times d_1}$ with singular values $\{\sigma_i\}$, the paper defines

$$L = \min(N,d_1), \quad q_i = \frac{\sigma_i}{\sum_{j=1}^{L} \sigma_j},$$

and

$$erank(A) = \exp\left(-\sum_{i=1}^{L} q_i \log q_i\right).$$

The stated range is from $1$ to $L$: low $erank$ means the token set lies in a few dominant directions, while high $H(p) = - \sum_i p_i \log p_i.$0 means the set is more dispersed in feature space (Baek et al., 1 Mar 2026).

These quantities are used jointly to characterize image complexity. The paper describes simple images as those with low attention entropy and low $H(p) = - \sum_i p_i \log p_i.$1, where evidence is concentrated in a few salient regions and important objects are easy to isolate. Complex images have higher entropy and higher $H(p) = - \sum_i p_i \log p_i.$2, indicating that useful evidence is spread across more regions and the token embeddings are more diverse. This characterization is then used to motivate adaptive pruning rules rather than static token-selection policies (Baek et al., 1 Mar 2026).

3. Empirical characterization of attention-based and diversity-based pruning

A major contribution of AgilePruner is its claim that many diversity-oriented pruning methods preserve substantially less feature diversity than their design rhetoric suggests. On POPE, for retained 64-token sets, the reported mean $H(p) = - \sum_i p_i \log p_i.$3 values are $H(p) = - \sum_i p_i \log p_i.$4 for PruMerge+, $H(p) = - \sum_i p_i \log p_i.$5 for VisionZip, $H(p) = - \sum_i p_i \log p_i.$6 for VisPruner, and $H(p) = - \sum_i p_i \log p_i.$7 for DivPrune. The paper uses these measurements to argue that nominally diversity-based selection does not necessarily produce highly diverse retained subsets in practice (Baek et al., 1 Mar 2026).

The work also links retained diversity to hallucination behavior. On CHAIR, where lower values are better and the metrics are defined as

$H(p) = - \sum_i p_i \log p_i.$8

attention-based methods are reported to be more conservative. FasterVLM obtains $H(p) = - \sum_i p_i \log p_i.$9 and $erank$0, while DivPrune reaches $erank$1 and $erank$2, and FPSPruner reaches $erank$3 and $erank$4. At the same time, diversity-based methods often show higher recall, meaning that they mention more ground-truth objects but also more nonexistent ones. The qualitative interpretation given in the paper is that diversity-oriented captions are broader and more speculative, whereas attention-based captions are narrower and more grounded (Baek et al., 1 Mar 2026).

The second major empirical result is that neither pruning family is universally superior. Attention-based pruning is more effective on simple images with concentrated evidence, while diversity-based pruning is better on complex images with distributed features. The paper reports that OCR and numerical or calculation tasks exhibit lower entropy and lower $erank$5 than POPE-style tasks, and that ScienceQA behaves more like the simple-image regime whereas POPE behaves more like the complex-image regime. This directly motivates a policy that varies the pruning behavior with image statistics rather than with a fixed global rule (Baek et al., 1 Mar 2026).

4. Adaptive pruning rule and mechanism

AgilePruner implements its conclusions through a deliberately minimal mechanism. Tokens are first sorted in descending order of attention score. The highest-ranked token is selected, and then other candidates whose cosine distance to that selected token is smaller than an adaptive threshold are pruned away. The process repeats with the next highest-ranked unpruned token until the target number of tokens has been selected (Baek et al., 1 Mar 2026).

Within this procedure, the threshold is the control variable that determines how strongly the algorithm favors diversity. A low threshold removes only very similar tokens and therefore remains closer to pure high-attention selection. A high threshold prunes more aggressively around already selected tokens, forcing the retained set to spread out more in feature space. The adaptive component comes from setting that threshold according to image complexity. The paper states that the threshold depends on the 1-based token rank $erank$6, the effective rank of the current image $erank$7, the average effective rank of the LLaVA training set $erank$8, and a factor of $erank$9, and that it is capped by a statistical upper bound $A \in \mathbb{R}^{N \times d_1}$0 (Baek et al., 1 Mar 2026).

The paper’s design principle is therefore explicit: simple, low-$A \in \mathbb{R}^{N \times d_1}$1 images should receive more attention-oriented selection, whereas complex, high-$A \in \mathbb{R}^{N \times d_1}$2 images should receive stronger diversity pressure. This same principle is also explored in an appendix as adaptive token count, where complex images retain more tokens and simple images are pruned more aggressively. Additional appendix experiments report that using entropy instead of $A \in \mathbb{R}^{N \times d_1}$3 for adaptive thresholding yields similar trends, although $A \in \mathbb{R}^{N \times d_1}$4 remains the main signal because it is said to better capture feature-space geometry (Baek et al., 1 Mar 2026).

AgilePruner is also presented as an explanatory device for hybrid methods more generally. The paper reports that applying its image-aware rule to existing hybrids such as VisPruner, BAT, and a direct FasterVLM+DivPrune mixture consistently improves benchmark performance, whereas the inverse rule degrades it. The stated implication is that hybridization alone is insufficient; the mixture has to match image complexity (Baek et al., 1 Mar 2026).

5. Experimental setup and reported performance

The reported main backbone is LLaVA-1.5-7B with deterministic decoding $A \in \mathbb{R}^{N \times d_1}$5 and fixed pretrained weights. Baselines include attention-based methods such as FastV, Sparse VLM, PyramidDrop, VisionZip, and VisPruner; diversity-based methods such as DivPrune; and hybrid or mixed strategies such as BAT and attention-diversity combinations. Evaluation is conducted on VQAv2, GQA, VizWiz, ScienceQA, TextVQA, POPE, MME, MMBench, and MMBench-CN, with additional hallucination evaluation on CHAIR and efficiency reporting in FLOPs, latency, and GPU memory (Baek et al., 1 Mar 2026).

The headline quantitative claim is that AgilePruner is best or near-best on the average across the nine benchmark datasets in the 128-, 64-, and 32-token regimes. At 64 tokens on LLaVA-1.5-7B, its average normalized score is reported as $A \in \mathbb{R}^{N \times d_1}$6, compared with $A \in \mathbb{R}^{N \times d_1}$7 for VisPruner and $A \in \mathbb{R}^{N \times d_1}$8 for DivPrune. The same general trend is reported for LLaVA-1.5-13B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B, which the paper uses to support the claim that the underlying empirical principle is model-agnostic (Baek et al., 1 Mar 2026).

Setting	Method	Reported result
LLaVA-1.5-7B, 64 tokens	AgilePruner	Average normalized score $A \in \mathbb{R}^{N \times d_1}$9
LLaVA-1.5-7B, 64 tokens	VisPruner	Average normalized score $\{\sigma_i\}$0
LLaVA-1.5-7B, 64 tokens	DivPrune	Average normalized score $\{\sigma_i\}$1
CHAIR, 64 tokens	AgilePruner	$\{\sigma_i\}$2, $\{\sigma_i\}$3, recall $\{\sigma_i\}$4, length $\{\sigma_i\}$5
CHAIR, 64 tokens	FasterVLM	$\{\sigma_i\}$6, $\{\sigma_i\}$7, recall $\{\sigma_i\}$8, length $\{\sigma_i\}$9
CHAIR, 64 tokens	DivPrune	$$L = \min(N,d_1), \quad q_i = \frac{\sigma_i}{\sum_{j=1}^{L} \sigma_j},$$0, $$L = \min(N,d_1), \quad q_i = \frac{\sigma_i}{\sum_{j=1}^{L} \sigma_j},$$1, recall $$L = \min(N,d_1), \quad q_i = \frac{\sigma_i}{\sum_{j=1}^{L} \sigma_j},$$2, length $$L = \min(N,d_1), \quad q_i = \frac{\sigma_i}{\sum_{j=1}^{L} \sigma_j},$$3

The CHAIR results are especially important because they expose the accuracy-grounding trade-off more directly than standard benchmarks. AgilePruner does not minimize hallucination as aggressively as the most conservative attention-based methods, but it reduces the hallucination penalty of diversity-based selection while retaining much of the recall. The paper describes this as an improved hallucination-coverage trade-off rather than a universal optimum on every grounding metric (Baek et al., 1 Mar 2026).

6. Interpretation, limitations, and terminological disambiguation

AgilePruner’s main interpretive claim is that visual token pruning should not be treated as a one-size-fits-all compression step. The optimal policy depends on whether visual evidence is concentrated or dispersed, and therefore on per-image statistics rather than only on global architectural preferences. This suggests a broader research direction in which token pruning is driven by sample-level structure and grounded reliability constraints, not only by token importance scores or set-dispersion heuristics (Baek et al., 1 Mar 2026).

The limitations are also explicit. The paper does not claim that a single adaptive rule solves all pruning cases. It notes that a nominally simple image may still contain many scattered objects, while a nominally complex image may contain one small localized cue that matters most. The method is therefore positioned as a strong empirical explanation of when attention, diversity, and hybrids tend to work, rather than as a final theory of token selection. The appendix-level observations that $$L = \min(N,d_1), \quad q_i = \frac{\sigma_i}{\sum_{j=1}^{L} \sigma_j},$$4 is stable under corruption and lightweight relative to full inference strengthen the practicality of the approach, but they do not eliminate the underlying ambiguity of difficult multimodal reasoning cases (Baek et al., 1 Mar 2026).

The name can also be misread because the supplied literature includes several distinct pruning research programs in robotics and agriculture. These include a learned hybrid vision/interaction controller for precision fruit tree pruning (You et al., 2021), an autonomous robot for pruning modern planar fruit trees (You et al., 2022), semantics-guided skeletonization for robotic pruning (You et al., 2021), a LiDAR-based procedure for automated pruning suggestion (Westling et al., 2021), and a broader review of autonomous robotic pruning in orchards and vineyards (Navone et al., 12 May 2025). Those works concern physical plant pruning, pruning-point determination, and orchard manipulation; they are terminologically related but conceptually separate from AgilePruner as a visual token pruning method for LVLMs.