---
title: 'HiPrune: Efficient Visual Token Pruning'
url: https://www.emergentmind.com/topics/hiprune
type: topic
---

# HiPrune: Efficient Visual Token Pruning

HiPrune is a training-free, model-agnostic visual token pruning framework for Vision-Language Models (VLMs) that exploits the hierarchical attention structure within ViT-based vision encoders. Its central claim is that middle layers attend to object-centric regions, whereas deep layers capture global contextual features; on that basis, HiPrune retains three classes of visual tokens—Anchor tokens, Buffer tokens, and Register tokens—before passing the shortened visual sequence to the multimodal stack. The method requires no retraining, does not depend on CLS tokens, and is evaluated on LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL, where it is reported to preserve up to 99.3% task accuracy with only 33.3% tokens and 99.5% accuracy with just 11.1% tokens, while reducing inference FLOPs and latency by up to \(9\times\) [2508.00553].

## 1. Problem setting and motivation

HiPrune addresses a specific efficiency bottleneck in modern VLMs: images are encoded into long sequences of visual tokens, and those sequences often dominate inference cost. The paper gives two concrete examples. In LLaVA-1.5, one image becomes 576 visual tokens. In dynamic-resolution systems such as Qwen2.5-VL, a high-resolution webpage image can require more than 10,000 tokens [2508.00553].

The motivation is not only sequence length but redundancy. The paper reports a pilot study showing that visual tokens have high cosine similarity with neighboring tokens, much more than text tokens. It also reports that randomly pruning 50% visual tokens causes a smaller performance drop than removing only 5% text tokens. This suggests that visual-token compression is a particularly favorable operating point for reducing VLM inference cost [2508.00553].

HiPrune is positioned against several prior families of token-reduction methods. Query-compression approaches based on modules such as Q-Former require extra training. Similarity/diversity-based merge or prune methods such as ToMe are described as having reduced interpretability, with gradual merging that may be less compatible with FlashAttention-style fast execution. Attention-based methods that rely on CLS tokens are architecture-sensitive because not all encoders have CLS tokens; SigLIP is given as the explicit example. Methods using text-image cross-attention from the language side, such as FastV, PDrop, and TRIM, depend on LLM-side text-image attention rather than the vision encoder’s internal hierarchy [2508.00553].

## 2. Hierarchical attention in vision encoders

The method rests on an empirical observation about depth-wise attention organization in vision encoders. The paper analyzes attention across layers using t-SNE of attention rankings, visualization of high-attention tokens, overlap with object segmentation masks, and dispersion statistics of selected tokens. The resulting picture is a depth hierarchy: shallow layers retain more noise and low-level patterns, middle layers become object-centric, and deep layers become globally contextual [2508.00553].

The strongest quantitative evidence is an IoU study between object segmentation masks and the top 10% high-attention tokens across layers. In Table 1, the middle layer \(L/2\) is normalized to \(1\times\) and consistently outperforms shallow and deep layers across CLIP-L, CLIP-B, SigLIP, SigLIP2, DeiT, and VJEPA2. The paper treats this as evidence that middle-layer attention is the most reliable source for selecting object-centric tokens [2508.00553].

The paper also emphasizes layer-wise continuity. Attention rankings evolve smoothly across neighboring layers rather than changing abruptly, and as layers deepen, high-attention tokens shift between semantic clusters. This suggests that the encoder builds a hierarchy rather than a set of unrelated per-layer saliency maps. HiPrune uses that hierarchy directly: middle-layer rankings identify object evidence, while deep-layer rankings identify tokens that act as global summaries.

## 3. Token taxonomy and selection mechanism

HiPrune prunes at the level of visual tokens. Let the final token matrix be
\[
T \in \mathbb{R}^{N \times d},
\]
where \(N\) is the number of image patches/tokens and \(d\) is the hidden dimension. For layer \(l\), token importance is aggregated from attention as
\[
a^{[l]} = \sum_{h=1}^{H}\sum_{n=1}^{N} A^{[l]}[h,n,:],
\]
and written as
\[
a^{[l]} = \left(a^{[l]}_1, a^{[l]}_2, \ldots, a^{[l]}_N\right) \in \mathbb{R}^N.
\]
Thus \(a^{[l]}_i\) is the aggregated attention score for token \(i\) at layer \(l\) [2508.00553].

HiPrune retains three token types.

**Anchor tokens** are the top-\(N_a\) tokens under attention in a selected middle/object layer \(l\). Their index set is
\[
I_A = \left\{\, i \mid \left|\{k \mid a^{[l]}_k > a^{[l]}_i\}\right| < N_a \right\}.
\]

**Buffer tokens** are spatial neighbors of anchors, added to preserve local continuity and reduce sensitivity to noisy attention maps. If there are \(p\) patches per row, the buffer set is
\[
I_B = \bigcup \{I_A - 1,\; I_A + 1,\; I_A - p,\; I_A + p\} \cap [0, p^2 - 1].
\]
This is the paper’s cross-shaped neighborhood: left, right, up, and down [2508.00553].

**Register tokens** are selected from the output/deep layer and are intended to preserve global contextual information. Excluding already selected anchor and buffer tokens, the remaining budget is filled by
\[
I_R = \left\{\, i \mid \left|\{j \mid a^{[L-1]}_j > a^{[L-1]}_i\}\right| < N' - |I_A \cup I_B| \right\},
\]
with
\[
i \notin I_A \cup I_B.
\]

The retained token vectors are then gathered from the encoder’s final representation space:
\[
T' = T[I_A \cup I_B \cup I_R, :] \in \mathbb{R}^{N' \times d}.
\]
This point is important: different layers are used to choose indices, but the actual output tokens forwarded downstream come from the final token matrix, so the VLM interface remains unchanged [2508.00553].

The method uses a target token budget \(N'\), an object layer \(l\), and an object proportion \(\alpha\). For the cross strategy, where one anchor expands to a five-token cluster including four buffers, the text states
\[
N_a = \left[ \frac{\alpha}{5} N' \right].
\]
For LLaVA-1.5-7B and LLaVA-NeXT-7B, the reported setting is \(l = 9\), \(\alpha = 0.1\). For Qwen, the reported setting is \(l = 16\), \(\alpha = 0.1\) [2508.00553].

## 4. Training-free integration and experimental protocol

HiPrune is training-free because it does not add trainable modules, does not fine-tune the VLM, and only uses attention maps already produced inside the vision encoder. The LLM is unchanged. Operationally, the method is inserted between the vision encoder output and the rest of the multimodal pipeline, where it shortens the visual token sequence before it enters the language model [2508.00553].

Its scope is narrower than the phrase “model-agnostic” may imply in isolation. The method assumes a ViT-based visual encoder, access to self-attention maps, and an image patch grid so that spatial adjacency is defined. Within that scope, the paper argues that HiPrune is more architecture-agnostic than CLS-dependent methods because it uses global aggregated attention rather than attention from a special token. The paper explicitly studies or claims generalization across CLIP, SigLIP, SigLIP2, DeiT, and VJEPA2 as vision encoders, and across LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL as downstream VLMs [2508.00553].

The primary evaluated models are LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen2.5-VL-3B-Instruct. Additional experiments in the supplement use LLaVA-1.5-13B, LLaVA-NeXT-13B, and Qwen2.5-VL-7B-Instruct. The benchmark suite includes GQA, SQA-IMG, VQAv2, MME, TextVQA, POPE, VizWiz, MMB, and MMB-CN. Most evaluations use LMMs-Eval; some LLaVA benchmark results are obtained using the public LLaVA codebase. FLOPs are computed with calflops, and experiments are run on one NVIDIA A100-PCIE (40G) [2508.00553].

## 5. Empirical performance and efficiency

The reported results show that HiPrune is strongest at moderate pruning ratios and remains competitive under more aggressive compression. The headline numbers reported in the abstract are supported by the main tables: 99.3% average performance with 33.3% tokens on LLaVA-1.5-7B, and 99.5% average performance with 11.1% tokens on LLaVA-NeXT-7B [2508.00553].

| Model | Retained tokens | Relative performance |
|---|---:|---:|
| LLaVA-1.5-7B | 192 tokens (33.3%) | 99.3% |
| LLaVA-1.5-7B | 128 tokens (22.2%) | 97.5% |
| LLaVA-1.5-7B | 64 tokens (11.1%) | 92.7% |
| LLaVA-NeXT-7B | 640 tokens (22.2%) | 102.6% |
| LLaVA-NeXT-7B | 320 tokens (11.1%) | 99.5% |
| LLaVA-NeXT-7B | 160 tokens (5.6%) | 92.5% |
| Qwen2.5-VL-3B-Instruct | 33.3% tokens | 98.6% |
| Qwen2.5-VL-3B-Instruct | 22.2% tokens | 97.1% |
| Qwen2.5-VL-3B-Instruct | 11.1% tokens | 92.8% |

On LLaVA-1.5-7B, HiPrune is best at 192 tokens and 128 tokens, and tied-best with VisionZip at 64 tokens, both at 92.7%. On LLaVA-NeXT-7B, HiPrune reports 102.6% of baseline average performance at 640 retained tokens and 99.5% at 320 retained tokens. At the most aggressive setting, 160 retained tokens, HiPrune remains competitive but is no longer the best: VisionZip reports 93.5% and TRIM 92.6%, versus HiPrune’s 92.5%. On Qwen2.5-VL-3B-Instruct, HiPrune exceeds both VisionZip and FastV at 33.3%, 22.2%, and 11.1% retention [2508.00553].

The efficiency analysis is concentrated on LLaVA-NeXT-7B. With 320 retained tokens, the paper reports 99.5% performance while using only 1.6% of FLOPs relative to the baseline setting in the table. With 160 retained tokens, it reports an \(8.8\times\) FLOPs reduction and a \(9.2\times\) prefill-time reduction, along with lower decode latency and VRAM, at the cost of only 7.5% performance loss. The abstract summarizes the broader claim as up to \(9\times\) FLOPs and latency reduction [2508.00553].

## 6. Ablations, limitations, and terminological boundaries

The ablation studies clarify which design choices matter. A comparison between CLS-token attention and global aggregated attention reports 99.8% versus 100.0%, favoring global aggregation while avoiding dependence on a special token. Token-type ablations on LLaVA-1.5-7B with 192 retained tokens report 97.9% without Register tokens, 99.7% without Buffer tokens, and 99.7% without Buffer plus Anchor tokens; the paper identifies removal of Register tokens as causing the largest degradation, which is consistent with the claim that deep-layer tokens preserve global context [2508.00553].

Buffer geometry is reported to be relatively insensitive so long as the neighborhood is sufficiently large. Table 5(c) reports Square(8) = 100.0%, Cross(4) = 100.0%, and Row(2) = 99.6%. The object-layer sweep in Figure 7 shows the best performance when anchors and buffers are selected from layers 5–9 in CLIP, aligning with the object-centric middle-layer hypothesis [2508.00553].

The limits of the method are also explicit. HiPrune is not directly applicable to non-ViT encoders without adaptation. Deployment requires access to internal attentions and token indexing on the image grid. The method also depends on hyperparameters such as the object layer \(l\), the object proportion \(\alpha\), and the neighborhood scheme. At the strongest compression settings, alternative methods can outperform it; the clearest example is LLaVA-NeXT at 5.6% tokens, where VisionZip slightly exceeds HiPrune [2508.00553].

The name should be distinguished from several unrelated pruning methods. **HFPrune** refers to “High-Fidelity Pruning for Large Language Models,” a first-order Taylor MLP-neuron pruning method for LLMs [2603.08083]. **HierarchicalPrune** refers to “HierarchicalPrune: Position-Aware Compression for Large-Scale Diffusion Models,” a compression framework for MMDiT diffusion models [2508.04663]. **HiPP-Prune** refers to “HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Models,” a plan-level structured pruning framework for VLM language backbones [2603.06270]. HiPrune, by contrast, is specifically a training-free visual token pruning method based on hierarchical attention in ViT-based vision encoders [2508.00553].

Source: https://www.emergentmind.com/topics/hiprune