HiPrune: Efficient Visual Token Pruning

• HiPrune is a training-free, model-agnostic method that prunes visual tokens in VLMs by exploiting hierarchical attention in ViT-based encoders.
• It classifies tokens into Anchor, Buffer, and Register types to preserve object-centric and global context while reducing inference costs.
• Empirical evaluations show that HiPrune maintains over 90% task accuracy with drastic reductions in tokens, FLOPs, and latency across diverse models.

HiPrune is a training-free, model-agnostic visual token pruning framework for Vision-LLMs (VLMs) that exploits the hierarchical attention structure within ViT-based vision encoders. Its central claim is that middle layers attend to object-centric regions, whereas deep layers capture global contextual features; on that basis, HiPrune retains three classes of visual tokens—Anchor tokens, Buffer tokens, and Register tokens—before passing the shortened visual sequence to the multimodal stack. The method requires no retraining, does not depend on CLS tokens, and is evaluated on LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL, where it is reported to preserve up to 99.3% task accuracy with only 33.3% tokens and 99.5% accuracy with just 11.1% tokens, while reducing inference FLOPs and latency by up to $9\times$ (Liu et al., 1 Aug 2025).

1. Problem setting and motivation

HiPrune addresses a specific efficiency bottleneck in modern VLMs: images are encoded into long sequences of visual tokens, and those sequences often dominate inference cost. The paper gives two concrete examples. In LLaVA-1.5, one image becomes 576 visual tokens. In dynamic-resolution systems such as Qwen2.5-VL, a high-resolution webpage image can require more than 10,000 tokens (Liu et al., 1 Aug 2025).

The motivation is not only sequence length but redundancy. The paper reports a pilot study showing that visual tokens have high cosine similarity with neighboring tokens, much more than text tokens. It also reports that randomly pruning 50% visual tokens causes a smaller performance drop than removing only 5% text tokens. This suggests that visual-token compression is a particularly favorable operating point for reducing VLM inference cost (Liu et al., 1 Aug 2025).

HiPrune is positioned against several prior families of token-reduction methods. Query-compression approaches based on modules such as Q-Former require extra training. Similarity/diversity-based merge or prune methods such as ToMe are described as having reduced interpretability, with gradual merging that may be less compatible with FlashAttention-style fast execution. Attention-based methods that rely on CLS tokens are architecture-sensitive because not all encoders have CLS tokens; SigLIP is given as the explicit example. Methods using text-image cross-attention from the language side, such as FastV, PDrop, and TRIM, depend on LLM-side text-image attention rather than the vision encoder’s internal hierarchy (Liu et al., 1 Aug 2025).

2. Hierarchical attention in vision encoders

The method rests on an empirical observation about depth-wise attention organization in vision encoders. The paper analyzes attention across layers using t-SNE of attention rankings, visualization of high-attention tokens, overlap with object segmentation masks, and dispersion statistics of selected tokens. The resulting picture is a depth hierarchy: shallow layers retain more noise and low-level patterns, middle layers become object-centric, and deep layers become globally contextual (Liu et al., 1 Aug 2025).

The strongest quantitative evidence is an IoU study between object segmentation masks and the top 10% high-attention tokens across layers. In Table 1, the middle layer $L/2$ is normalized to $1\times$ and consistently outperforms shallow and deep layers across CLIP-L, CLIP-B, SigLIP, SigLIP2, DeiT, and VJEPA2. The paper treats this as evidence that middle-layer attention is the most reliable source for selecting object-centric tokens (Liu et al., 1 Aug 2025).

The paper also emphasizes layer-wise continuity. Attention rankings evolve smoothly across neighboring layers rather than changing abruptly, and as layers deepen, high-attention tokens shift between semantic clusters. This suggests that the encoder builds a hierarchy rather than a set of unrelated per-layer saliency maps. HiPrune uses that hierarchy directly: middle-layer rankings identify object evidence, while deep-layer rankings identify tokens that act as global summaries.

3. Token taxonomy and selection mechanism

HiPrune prunes at the level of visual tokens. Let the final token matrix be

$T \in \mathbb{R}^{N \times d},$

where $N$ is the number of image patches/tokens and $d$ is the hidden dimension. For layer $l$, token importance is aggregated from attention as

$$a^{[l]} = \sum_{h=1}^{H}\sum_{n=1}^{N} A^{[l]}[h,n,:],$$

and written as

$$a^{[l]} = \left(a^{[l]}_1, a^{[l]}_2, \ldots, a^{[l]}_N\right) \in \mathbb{R}^N.$$

Thus $a^{[l]}_i$ is the aggregated attention score for token $L/2$0 at layer $L/2$1 (Liu et al., 1 Aug 2025).

HiPrune retains three token types.

Anchor tokens are the top-$L/2$2 tokens under attention in a selected middle/object layer $L/2$3. Their index set is

$L/2$4

Buffer tokens are spatial neighbors of anchors, added to preserve local continuity and reduce sensitivity to noisy attention maps. If there are $L/2$5 patches per row, the buffer set is

$L/2$6

This is the paper’s cross-shaped neighborhood: left, right, up, and down (Liu et al., 1 Aug 2025).

Register tokens are selected from the output/deep layer and are intended to preserve global contextual information. Excluding already selected anchor and buffer tokens, the remaining budget is filled by

$L/2$7

with

$L/2$8

The retained token vectors are then gathered from the encoder’s final representation space: $L/2$9 This point is important: different layers are used to choose indices, but the actual output tokens forwarded downstream come from the final token matrix, so the VLM interface remains unchanged (Liu et al., 1 Aug 2025).

The method uses a target token budget $1\times$0, an object layer $1\times$1, and an object proportion $1\times$2. For the cross strategy, where one anchor expands to a five-token cluster including four buffers, the text states

$1\times$3

For LLaVA-1.5-7B and LLaVA-NeXT-7B, the reported setting is $1\times$4, $1\times$5. For Qwen, the reported setting is $1\times$6, $1\times$7 (Liu et al., 1 Aug 2025).

4. Training-free integration and experimental protocol

HiPrune is training-free because it does not add trainable modules, does not fine-tune the VLM, and only uses attention maps already produced inside the vision encoder. The LLM is unchanged. Operationally, the method is inserted between the vision encoder output and the rest of the multimodal pipeline, where it shortens the visual token sequence before it enters the LLM (Liu et al., 1 Aug 2025).

Its scope is narrower than the phrase “model-agnostic” may imply in isolation. The method assumes a ViT-based visual encoder, access to self-attention maps, and an image patch grid so that spatial adjacency is defined. Within that scope, the paper argues that HiPrune is more architecture-agnostic than CLS-dependent methods because it uses global aggregated attention rather than attention from a special token. The paper explicitly studies or claims generalization across CLIP, SigLIP, SigLIP2, DeiT, and VJEPA2 as vision encoders, and across LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL as downstream VLMs (Liu et al., 1 Aug 2025).

The primary evaluated models are LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen2.5-VL-3B-Instruct. Additional experiments in the supplement use LLaVA-1.5-13B, LLaVA-NeXT-13B, and Qwen2.5-VL-7B-Instruct. The benchmark suite includes GQA, SQA-IMG, VQAv2, MME, TextVQA, POPE, VizWiz, MMB, and MMB-CN. Most evaluations use LMMs-Eval; some LLaVA benchmark results are obtained using the public LLaVA codebase. FLOPs are computed with calflops, and experiments are run on one NVIDIA A100-PCIE (40G) (Liu et al., 1 Aug 2025).

5. Empirical performance and efficiency

The reported results show that HiPrune is strongest at moderate pruning ratios and remains competitive under more aggressive compression. The headline numbers reported in the abstract are supported by the main tables: 99.3% average performance with 33.3% tokens on LLaVA-1.5-7B, and 99.5% average performance with 11.1% tokens on LLaVA-NeXT-7B (Liu et al., 1 Aug 2025).

Model	Retained tokens	Relative performance
LLaVA-1.5-7B	192 tokens (33.3%)	99.3%
LLaVA-1.5-7B	128 tokens (22.2%)	97.5%
LLaVA-1.5-7B	64 tokens (11.1%)	92.7%
LLaVA-NeXT-7B	640 tokens (22.2%)	102.6%
LLaVA-NeXT-7B	320 tokens (11.1%)	99.5%
LLaVA-NeXT-7B	160 tokens (5.6%)	92.5%
Qwen2.5-VL-3B-Instruct	33.3% tokens	98.6%
Qwen2.5-VL-3B-Instruct	22.2% tokens	97.1%
Qwen2.5-VL-3B-Instruct	11.1% tokens	92.8%

On LLaVA-1.5-7B, HiPrune is best at 192 tokens and 128 tokens, and tied-best with VisionZip at 64 tokens, both at 92.7%. On LLaVA-NeXT-7B, HiPrune reports 102.6% of baseline average performance at 640 retained tokens and 99.5% at 320 retained tokens. At the most aggressive setting, 160 retained tokens, HiPrune remains competitive but is no longer the best: VisionZip reports 93.5% and TRIM 92.6%, versus HiPrune’s 92.5%. On Qwen2.5-VL-3B-Instruct, HiPrune exceeds both VisionZip and FastV at 33.3%, 22.2%, and 11.1% retention (Liu et al., 1 Aug 2025).

The efficiency analysis is concentrated on LLaVA-NeXT-7B. With 320 retained tokens, the paper reports 99.5% performance while using only 1.6% of FLOPs relative to the baseline setting in the table. With 160 retained tokens, it reports an $1\times$8 FLOPs reduction and a $1\times$9 prefill-time reduction, along with lower decode latency and VRAM, at the cost of only 7.5% performance loss. The abstract summarizes the broader claim as up to $T \in \mathbb{R}^{N \times d},$0 FLOPs and latency reduction (Liu et al., 1 Aug 2025).

6. Ablations, limitations, and terminological boundaries

The ablation studies clarify which design choices matter. A comparison between CLS-token attention and global aggregated attention reports 99.8% versus 100.0%, favoring global aggregation while avoiding dependence on a special token. Token-type ablations on LLaVA-1.5-7B with 192 retained tokens report 97.9% without Register tokens, 99.7% without Buffer tokens, and 99.7% without Buffer plus Anchor tokens; the paper identifies removal of Register tokens as causing the largest degradation, which is consistent with the claim that deep-layer tokens preserve global context (Liu et al., 1 Aug 2025).

Buffer geometry is reported to be relatively insensitive so long as the neighborhood is sufficiently large. Table 5(c) reports Square(8) = 100.0%, Cross(4) = 100.0%, and Row(2) = 99.6%. The object-layer sweep in Figure 1 shows the best performance when anchors and buffers are selected from layers 5–9 in CLIP, aligning with the object-centric middle-layer hypothesis (Liu et al., 1 Aug 2025).

The limits of the method are also explicit. HiPrune is not directly applicable to non-ViT encoders without adaptation. Deployment requires access to internal attentions and token indexing on the image grid. The method also depends on hyperparameters such as the object layer $T \in \mathbb{R}^{N \times d},$1, the object proportion $T \in \mathbb{R}^{N \times d},$2, and the neighborhood scheme. At the strongest compression settings, alternative methods can outperform it; the clearest example is LLaVA-NeXT at 5.6% tokens, where VisionZip slightly exceeds HiPrune (Liu et al., 1 Aug 2025).

The name should be distinguished from several unrelated pruning methods. HFPrune refers to “High-Fidelity Pruning for LLMs,” a first-order Taylor MLP-neuron pruning method for LLMs (Zhu et al., 9 Mar 2026). HierarchicalPrune refers to “HierarchicalPrune: Position-Aware Compression for Large-Scale Diffusion Models,” a compression framework for MMDiT diffusion models (Kwon et al., 6 Aug 2025). HiPP-Prune refers to “HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-LLMs,” a plan-level structured pruning framework for VLM language backbones (Bai et al., 6 Mar 2026). HiPrune, by contrast, is specifically a training-free visual token pruning method based on hierarchical attention in ViT-based vision encoders (Liu et al., 1 Aug 2025).