Pyramid Token Pruning (PTP) for LVLMs

• Pyramid Token Pruning (PTP) is a training-free hierarchical visual token pruning framework that optimizes inference in high-resolution LVLMs using region-level, patch token, and instruction-guided saliency.
• It employs a coarse-to-fine approach where global similarity guides region token budgets, and bottom-up attention and top-down instructions rank tokens for efficient multimodal processing.
• PTP achieves significant computational savings—reducing FLOPs by over 30% with minimal accuracy loss—demonstrating robust performance in high-resolution vision-language tasks.

Searching arXiv for the core PTP paper and closely related token-pruning work to ground the article in current literature. arXiv.search Pyramid Token Pruning (PTP) is a training-free visual token pruning framework for high-resolution large vision-LLMs (LVLMs) that combines region-level saliency, token-level saliency, and instruction-guided importance to retain the most informative visual tokens before they are fed to the LLM. It is inserted as a plug-and-play module between the vision encoder and the LLM, and it is called “pyramid” because pruning proceeds in a hierarchical coarse-to-fine manner: first across image regions, then across patch tokens within each region, and finally under top-down modulation from the user instruction (Liang et al., 19 Sep 2025).

1. Problem setting and motivation

PTP addresses a specific inefficiency of high-resolution LVLMs: modern systems often partition an image into multiple local sub-images together with a global thumbnail, encode each tile independently, and then concatenate the resulting visual tokens into a long sequence for multimodal inference. In this regime, each image crop yields a full patch-token sequence, the number of crops scales with image size or aspect ratio, and a global image is often processed in addition to all local crops. The result is a large increase in inference FLOPs, latency, GPU memory usage, and a reduction in throughput (Liang et al., 19 Sep 2025).

The framework is motivated by the claim that only a small subset of visual tokens meaningfully influences the final answer. The PTP paper cites the observation from LLaVA-1.5 that after a few layers, average attention to image tokens is only about $0.2\%$ of that on text tokens, which it interprets as evidence of substantial redundancy in the visual prefix (Liang et al., 19 Sep 2025). This makes token pruning especially relevant in multi-crop LVLMs, where visual sequence length is inflated by tiling rather than by a single dense image encoding.

The “pyramid” designation refers to a hierarchical allocation policy rather than to repeated layer-by-layer pruning inside the vision encoder. PTP first decides which image regions deserve larger token budgets, then selects salient patch tokens within those regions, and then further prioritizes visual tokens that are relevant to the current instruction. This suggests a coarse-to-fine attention structure: broad spatial allocation at the region level, fine-grained visual saliency at the token level, and top-down query conditioning at the instruction level (Liang et al., 19 Sep 2025).

2. Hierarchical pruning pipeline

The input image is partitioned into an $m \times n$ grid of local regions, and a global thumbnail of the whole image is also generated. This yields $m \cdot n + 1$ image tiles in total: $m \cdot n$ local crops and one global image. In the InternVL2-2B implementation, each tile is resized to resolution $448$ before visual encoding (Liang et al., 19 Sep 2025).

Each sub-image and the global thumbnail are independently fed through the vision encoder. For each tile, the encoder outputs a sequence of patch tokens and a global CLS token. Denoting the CLS embedding of local region $i$ by $\mathbf{cls}^{(i)}$ and the CLS embedding of the global image by $\mathbf{cls}^{(g)}$, PTP defines region-level importance as cosine similarity: $$a_i = \cos\left(\mathbf{cls}^{(i)},\ \mathbf{cls}^{(g)}\right).$$ A region with larger similarity to the global representation is treated as more semantically central and is assigned a larger token budget (Liang et al., 19 Sep 2025).

Local-region budgets are then allocated by

$R_i = (1-r) \cdot T \cdot \mathrm{softmax}(a_i),$

where $m \times n$0 is the desired overall pruning ratio and $m \times n$1 is the total number of visual tokens. The paper further states that the global thumbnail receives a separate retained budget

$m \times n$2

The paper itself notes that this global-budget expression is somewhat unusual relative to the local-region budget rule, but it is the stated formulation (Liang et al., 19 Sep 2025).

After region budgets are fixed, token ranking is performed within each region. PTP extracts self-attention statistics from a selected vision encoder layer $m \times n$3, using CLS-to-patch attention as a bottom-up token saliency signal. It then augments this with an instruction-guided score derived from attention in the second transformer block of the LLM after visual tokens have been projected into the language-model space and concatenated with the textual instruction (Liang et al., 19 Sep 2025).

The pipeline is therefore sequential and hierarchical rather than flat: region importance determines how many tokens are kept from each region, while token saliency and instruction relevance determine which tokens are kept within those regional budgets. The paper describes this as a training-free, task-aware, and coarse-to-fine pruning strategy inspired by human visual attention (Liang et al., 19 Sep 2025).

3. Bottom-up and top-down importance signals

Within region $m \times n$4, PTP defines bottom-up token importance for patch token $m \times n$5 using CLS-to-patch self-attention at vision encoder layer $m \times n$6: $m \times n$7 The equation in the paper contains a formatting typo, but the surrounding text states that the intended meaning is the attention coefficient from CLS to patch $m \times n$8 at layer $m \times n$9 (Liang et al., 19 Sep 2025).

Instruction-guided importance is extracted from the second LLM block. For each visual token $m \cdot n + 1$0,

$m \cdot n + 1$1

where $m \cdot n + 1$2 is the set of instruction token indices. The appendix also considers the mean alternative

$m \cdot n + 1$3

but the paper reports that max outperforms mean on POPE and MME, arguing that mean can be diluted by irrelevant instruction words (Liang et al., 19 Sep 2025).

The final fused token score is

$m \cdot n + 1$4

with $m \cdot n + 1$5. The recommended default is $m \cdot n + 1$6. Scores $m \cdot n + 1$7, $m \cdot n + 1$8, and $m \cdot n + 1$9 are normalized to the $m \cdot n$0 range before fusion so that their magnitudes are comparable, although the exact normalization formula is not provided (Liang et al., 19 Sep 2025).

The final pruning rule is straightforward. For each region $m \cdot n$1, tokens are ranked by $m \cdot n$2, and the top-$m \cdot n$3 tokens are retained. For the global thumbnail, an analogous ranking is used and the top-$m \cdot n$4 tokens are kept. The paper studies multiple vision encoder layers and reports that layer $m \cdot n$5 gives the best overall trade-off: earlier layers are better for OCR-like tasks, deeper layers can help scene-level reasoning, and middle layers provide the best general-purpose default (Liang et al., 19 Sep 2025).

This scoring decomposition is central to the identity of PTP. Region-level saliency acts as a coarse bottom-up budget allocator; token-level saliency identifies visually important patches within each region; instruction-guided attention introduces a top-down task signal. The paper emphasizes that these three signals are not treated symmetrically: region importance determines token budgets across regions, whereas bottom-up token saliency and instruction-guided importance are fused additively for intra-region token ranking (Liang et al., 19 Sep 2025).

4. Architectural interface and reported performance

PTP is explicitly inserted between the vision encoder and the LLM. In the reported implementation on InternVL2-2B, the vision encoder is InternViT-300M, the projector is a two-layer MLP, and the LLM is InternLM2-1.8B. The method relies on vision encoder CLS embeddings, vision self-attention, and early LLM instruction-to-vision attention, but it does not require architecture modification, additional supervision, fine-tuning of the LVLM, or external saliency models (Liang et al., 19 Sep 2025).

Its computational rationale is that visual tokens are pruned before they burden the LLM sequence excessively. Because transformer cost rises rapidly with sequence length, shortening the visual token sequence reduces LLM FLOPs, memory, and latency. The paper characterizes the extra overhead as low because PTP reuses existing CLS embeddings and attention maps, although the main quantitative table reports FLOPs rather than a full latency table (Liang et al., 19 Sep 2025).

Under the main setting $m \cdot n$6, the paper reports for InternVL2-2B + PTP an average accuracy retention of $m \cdot n$7, FLOPs of $m \cdot n$8 T, and a baseline FLOPs figure of $m \cdot n$9 T. This corresponds to a $448$0 FLOPs reduction with only $448$1 average accuracy loss (Liang et al., 19 Sep 2025). In the same table, Random at $448$2 gives $448$3 average retention and $448$4 T FLOPs, FastV gives $448$5 and $448$6 T, and GSearch gives $448$7 and $448$8 T, positioning PTP above these baselines on both accuracy and FLOPs under the reported setting (Liang et al., 19 Sep 2025).

Task-wise, the paper highlights Chart-QA at $448$9, POPE at $i$0, MM-Star at $i$1, and MME at $i$2, noting that on some comprehensive tasks pruning may even improve performance by removing noisy or redundant tokens (Liang et al., 19 Sep 2025). The ablation study further attributes an average decline of $i$3 to removing region allocation, $i$4 to removing region allocation together with bottom-up token scoring, $i$5 to removing region allocation together with instruction-guided scoring, and $i$6 to random pruning, which the paper interprets as strong evidence for the importance of instruction-aware pruning (Liang et al., 19 Sep 2025).

The fusion-weight ablation shows that $i$7 is best overall for general VQA and comprehensive reasoning, smaller $i$8 values such as $i$9 work better on OCR tasks such as Text-VQA and Chart-QA, and larger $\mathbf{cls}^{(i)}$0 values help RealWorldQA (Liang et al., 19 Sep 2025). This indicates that no single fixed balance between bottom-up and top-down signals is universally optimal.

5. Position within the token-reduction literature

PTP belongs to a broader line of work on hierarchical or progressive token reduction, but its mechanism differs from several adjacent formulations. In vision transformers, “Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers” argues that aggressive layerwise token reduction is limited not only by importance scoring but also by irreversible information loss from hard deletion, and proposes to squeeze information from pruned tokens into retained tokens instead of discarding them outright (Wei et al., 2023). This establishes a contrast: PTP performs region- and instruction-aware hard selection before the LLM, whereas TPS addresses information preservation inside progressive ViT pruning.

“Token Fusion: Bridging the Gap between Token Pruning and Token Merging” proposes a hybrid depth-aware strategy in which earlier layers use pruning-oriented reduction and later layers use merge-oriented reduction, motivated by increasing functional linearity with depth (Kim et al., 2023). That work suggests that hierarchical token reduction may need to vary not only token counts but also the reduction operator itself. PTP does not mix pruning and merging; it remains a pruning framework, but its coarse-to-fine region budget allocation is similarly hierarchical.

The relation to strict “pyramid” execution is also nontrivial. “HiPrune: Training-Free Visual Token Pruning via Hierarchical Attention in Vision-LLMs” exploits middle-layer object-centric attention and deep-layer global attention, but does so in a one-shot post-encoder token selection policy rather than progressive layer-by-layer reduction during the encoder forward pass (Liu et al., 1 Aug 2025). PTP is likewise hierarchical in saliency sources, yet its hierarchy is defined over regions, tokens, and instruction relevance rather than over progressive encoder-side token shrinkage.

By contrast, “HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit” is a more literal in-LLM pyramid pruning method: visual tokens are injected only when active fusion begins, pruned in a concave middle-layer schedule, and then completely removed at an early-exit layer (Wu et al., 27 Feb 2026). “Towards Joint Quantization and Token Pruning of Vision-LLMs” further interprets progressive pruning as a monotone, calibration-driven layer-wise keep-ratio schedule under W4A4KV4 deployment, with QUOTA producing a non-increasing offline pruning recipe (Li et al., 19 Apr 2026). These works make the contrast explicit: PTP is hierarchical in scale and signal type, whereas HiDrop and QUOTA are hierarchical in layer-wise execution.

The broader idea of combining coarse-to-fine region routing with query-conditioned selection also appears in remote sensing LVLMs. “When Large Vision-LLM Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruning” uses a Dynamic Image Pyramid and a Region Focus Module to perform text-guided tile selection and token pruning over extremely large images (Luo et al., 10 Mar 2025). This suggests that PTP is part of a larger family of coarse-to-fine multimodal token selection methods in which spatial hierarchy and task conditioning are jointly used to control visual-token budgets.

6. Limitations, caveats, and open directions

The PTP paper explicitly identifies several limitations. A fixed $\mathbf{cls}^{(i)}$1 is suboptimal because different tasks prefer different balances between bottom-up saliency and top-down instruction guidance. Architectural validation is limited: although the method is described as model-agnostic and easy to integrate into existing LVLM pipelines, experimental validation is mainly on InternVL2-2B (Liang et al., 19 Sep 2025). Instruction guidance can also be unreliable; the paper notes that misleading prompt tokens may corrupt top-down pruning on tasks such as POPE, which is why lower $\mathbf{cls}^{(i)}$2 can work better there (Liang et al., 19 Sep 2025).

A further limitation is structural rather than empirical. PTP is hierarchical in scale, but it is not a multi-stage recursive pruning schedule over many rounds inside the vision encoder or LLM. This suggests that its “pyramid” is coarse-to-fine in the sense of region allocation and token ranking, not in the sense of a monotone per-layer token-count law. A plausible implication is that PTP occupies an intermediate position between one-shot hierarchical token selection and fully progressive layer-wise pyramid pruning.

Adjacent literature also sharpens the practical caution around aggressive compression. In ViTs, TPS argues that dropped tokens continue to carry exclusive information even when ranked low by the pruning policy (Wei et al., 2023). In segmentation, “When Token Compression Breaks” reports that token compression is highly effective at mild reductions but degrades sharply when compression becomes severe, while structural pruning exhibits a smoother degradation curve and a prune-then-merge strategy gives a better accuracy-robustness trade-off at high compression (Nguyen et al., 2 Jul 2026). This suggests that PTP’s strong reported trade-off at $\mathbf{cls}^{(i)}$3 should not be generalized automatically to arbitrarily severe compression regimes.

Within those bounds, PTP’s distinctive contribution is clear. It formalizes a training-free visual token pruning rule for high-resolution LVLMs in which global-vs-local CLS similarity decides regional token budgets, CLS-to-patch attention ranks local visual evidence, and early LLM instruction-to-vision attention preserves query-relevant content (Liang et al., 19 Sep 2025). In the current literature, that combination makes PTP a representative instance of hierarchical, bottom-up plus top-down visual token pruning for high-resolution multimodal inference.