Position-Reweighted Visual Token Pruning

• The paper introduces position-reweighted pruning techniques that adjust token retention using spatial cues to counteract positional bias and reduce computational cost.
• It employs methods like zonal budgeting, coordinate anchors, and adaptive score correction to preserve spatial structure and improve grounding and other tasks.
• Empirical results demonstrate significant speedups and maintained performance, with reductions in TFLOPs and latency even under aggressive token compression.

Position-reweighted visual token pruning denotes a family of efficiency methods for vision-LLMs and multimodal LLMs in which token retention is not treated as a purely position-agnostic ranking problem. Instead, pruning decisions are explicitly adjusted by spatial zones, sequence position, local adjacency, implicit coordinate anchors, or preserved position IDs, in order to counteract positional bias and maintain spatial fidelity under aggressive compression. This line of work emerged from a shared observation: attention-only or similarity-only pruning can over-retain tokens from biased regions of the visual sequence, while spatially critical evidence may be lost even when its semantic content is indispensable (Zhao et al., 25 Aug 2025, Duan et al., 13 Nov 2025).

1. Formal setting and computational rationale

In the standard MLLM pipeline, a vision encoder such as CLIP ViT or SigLIP converts an image into patch tokens

$V = \{v_1, \dots, v_N\},$

with each $v_i \in \mathbb{R}^d$, after which a projector maps them into the LLM’s embedding space and concatenates them with text tokens. Visual token pruning is then formulated as selecting a subset $V' \subset V$ with $|V'| = k \ll N$ to reduce computation in the decoder (Duan et al., 13 Nov 2025).

The computational pressure comes from quadratic self-attention. One formulation used for decoder cost is

$$\text{TFLOPs} \approx L \cdot (2N^2 d + 4Nd^2 + 3N d m),$$

where $L$ is the number of LLM layers, $N$ the number of visual tokens, $d$ hidden size, and $m$ FFN intermediate size. The dominant term $2N^2 d$ makes token count reduction directly valuable. On LLaVA-1.5-7B, pruning from 576 to 192 tokens reduces TFLOPs from 3.82 to 1.25 and latency from 194.2 ms to 90.8 ms; on LLaVA-NeXT-7B, reducing from 2880 to 320 tokens lowers TFLOPs from 20.83 to 2.10 and latency from 576.9 ms to 113.4 ms (Duan et al., 13 Nov 2025).

A useful synthesis is that position-reweighted pruning modifies the usual objective “which tokens are important?” into a two-part question: which spatial locations deserve budget, and which tokens within those locations should survive. This distinction is explicit in later zonal and coordinate-aware methods, but it is already implicit in earlier analyses of recency bias, local redundancy, and spatial reasoning failures.

2. Positional pathologies that motivate reweighting

The most direct motivation comes from recency bias in attention-based pruning. In VLMs where image patches are flattened in raster order, later sequence positions often correspond to lower image regions. PoRe reports that the average text-visual attention score increases with token index, so pruning by raw last-token attention disproportionately retains tokens from the image bottom; its visualizations show bottom-heavy attention maps before correction and more balanced maps after reweighting (Zhao et al., 25 Aug 2025).

AdaptPrune reports a closely related phenomenon. Using 1000 samples from Nocaps and TextVQA, it finds that the average attention distribution over image patches exhibits stable motifs per layer and that higher attention is systematically biased to tokens at the end of the image sequence. The paper also shows that top-attention tokens are often adjacent patches with highly similar content, so attention-only pruning simultaneously amplifies positional bias and redundancy clustering (Luan et al., 11 Mar 2025).

GridPrune makes the same critique at the level of global Top-$v_i \in \mathbb{R}^d$0 selection. It formalizes the standard paradigm as choosing tokens with the largest scalar importance scores, but shows that methods such as FastV exhibit a “massive spike” at the end of the visual sequence. The argument is that global ranking has no notion of zones or coverage, so it can over-sample a few regions and miss small but critical objects (Duan et al., 13 Nov 2025).

Position sensitivity is even sharper in grounding. Grounding-Aware Token Pruning shows that on RefCOCO validation, LLaVA drops from 56.14% to 15.34% after pruning, and identifies misaligned position IDs as the primary cause. The same work reports that, for MiniGPTv2, CLS-visual pruning at 50% drives RefCOCO validation from 88.69% to 2.73%. Its central claim is that both the order and value of position IDs are crucial for grounding tasks (Chien et al., 27 Jun 2025).

A different but compatible diagnosis appears in IVC-Prune. That work argues that LVLMs implicitly establish visual coordinate systems through RoPE, and that specific token positions function as implicit visual coordinates needed for spatial reasoning. Under this view, certain positions are not merely likely to be useful; they are coordinate anchors whose removal disproportionately harms grounding and spatial queries (Sun et al., 3 Feb 2026).

3. Direct score correction and multi-cue position-aware ranking

The simplest explicit position-reweighting scheme is PoRe. It starts from the standard attention score

$v_i \in \mathbb{R}^d$1

where $v_i \in \mathbb{R}^d$2 is the query vector of the last text token and $v_i \in \mathbb{R}^d$3 is the key of visual token $v_i \in \mathbb{R}^d$4. PoRe decomposes this into a content-agnostic positional bias $v_i \in \mathbb{R}^d$5 and a desired content-only term $v_i \in \mathbb{R}^d$6,

$v_i \in \mathbb{R}^d$7

with $v_i \in \mathbb{R}^d$8 estimated by averaging attention scores over a dataset and fitting the resulting trend with a parametric exponential function. The reweighted scores $v_i \in \mathbb{R}^d$9 are then used in place of $V' \subset V$0 for pruning (Zhao et al., 25 Aug 2025).

AdaptPrune adopts a richer multi-cue formulation. It first applies a Gaussian correction centered at the image midpoint,

$V' \subset V$1

then performs adaptive non-maximum suppression using both spatial distance and token similarity. With Euclidean patch distance $V' \subset V$2 and cosine similarity between key vectors, it defines

$V' \subset V$3

and updates scores by

$V' \subset V$4

This makes position enter twice: once as a prior correcting biased attention, and again as a suppression factor that discourages redundant local clusters (Luan et al., 11 Mar 2025).

These direct score-correction approaches have concrete empirical effects. On LLaVA-1.5-7B at 90% pruning, PoRe improves over FastV from 65.2 to 70.1 on VQA$V' \subset V$5, from 51.1 to 53.9 on GQA, from 56.8 to 67.5 on POPE, and from 1227.3 to 1393.7 on MME, while keeping the same FLOPs as FastV (Zhao et al., 25 Aug 2025). Under the same 90% pruning regime, AdaptPrune on LLaVA-1.5-7B raises Nocaps CIDEr from 74.2 to 99.3 and GQA from 51.0 to 58.0 relative to FastV, showing that positional correction and redundancy suppression remain effective even under extreme compression (Luan et al., 11 Mar 2025).

4. Zonal budgeting, coordinate anchors, and protected spatial structure

A more structured formulation appears in GridPrune. Its central idea is a two-stage “guide-globally, select-locally” system. For each token, it computes text-conditional relevance

$V' \subset V$6

normalizes it to $V' \subset V$7, combines it with intrinsic visual saliency $V' \subset V$8,

$V' \subset V$9

and then partitions the image into zones. For a zone $|V'| = k \ll N$0, it forms aggregate relevance

$|V'| = k \ll N$1

converts these into a budget distribution

$|V'| = k \ll N$2

and assigns integer budgets $|V'| = k \ll N$3 under a per-zone capacity. Local Top-$|V'| = k \ll N$4 selection then chooses tokens within each zone according to $|V'| = k \ll N$5. This makes position an explicit budget variable rather than a side effect of token ranking (Duan et al., 13 Nov 2025).

GridPrune’s ablations argue that this spatial budgeting is not incidental. On LLaVA-1.5-7B with 192 tokens and $|V'| = k \ll N$6, a global block size of 24 gives 97.6% average performance, while progressively finer grids improve results, peaking at 98.6% with block size 2. On LLaVA-NeXT-7B, the method retains 96.98% of full performance at 320 tokens, or 11.1% of the original 2880-token input, and is 2.34% higher than DivPrune at the same retention (Duan et al., 13 Nov 2025).

IVC-Prune reinterprets position from the standpoint of RoPE geometry. For a position $|V'| = k \ll N$7, it examines when the RoPE rotation matrix is close to the identity or to a $|V'| = k \ll N$8 rotation, using

$|V'| = k \ll N$9

and defines implicit visual coordinates as the top positions under these scores. The final retained set is the union of IVC tokens and foreground tokens found through a two-stage value-vector similarity process. The method reports approximately 50% token reduction while maintaining $$\text{TFLOPs} \approx L \cdot (2N^2 d + 4Nd^2 + 3N d m),$$0 of the original performance and even achieving improvements on several benchmarks (Sun et al., 3 Feb 2026).

A related, but more general, inference is that later position-reweighted methods increasingly treat spatial structure as a protected object: zones receive budgets, coordinate anchors are always retained, and local selection is constrained by explicit spatial partitions rather than by global scalar ranking alone.

5. Position IDs, native order, and spatio-temporal locality

Some of the strongest evidence for position-aware pruning concerns not score computation but sequence structure. GAP shows that naive pruning often fails because retained tokens are reindexed as if they formed a new contiguous sequence. Its correction is simple: if pruning selects indices $$\text{TFLOPs} \approx L \cdot (2N^2 d + 4Nd^2 + 3N d m),$$1, then the model should use the original position IDs $$\text{TFLOPs} \approx L \cdot (2N^2 d + 4Nd^2 + 3N d m),$$2 rather than newly assigned positions. This recovers RefCOCO validation on LLaVA-v1.5-7B from 15.34% to 51.42% under 50% CLS-visual pruning, while requiring no additional training, memory, or computational overhead (Chien et al., 27 Jun 2025).

AnchorPrune, although focused on relevance-anchored contextual expansion rather than explicit position weighting, also reports that restoring native order matters. With identical retained subsets, raw selection order yields MME-OCR 132.5 and OCRBench 26.9, whereas raster order raises these to 140.0 and 28.9. This suggests that sequence order remains a meaningful spatial signal even after token selection has been decided (Oh et al., 8 Jul 2026).

For streaming video, StreamingAssistant introduces an explicitly local spatial redundancy metric, Maximum Similarity to Spatially Adjacent Video Tokens: $$\text{TFLOPs} \approx L \cdot (2N^2 d + 4Nd^2 + 3N d m),$$3 To avoid the bidirectional dependency between pruning and redundancy, it applies a checkerboard mask

$$\text{TFLOPs} \approx L \cdot (2N^2 d + 4Nd^2 + 3N d m),$$4

and prunes by

$$\text{TFLOPs} \approx L \cdot (2N^2 d + 4Nd^2 + 3N d m),$$5

This guarantees that no two pruned tokens are spatially adjacent, preserves spatial structure, and combines naturally with temporal redundancy pruning through $$\text{TFLOPs} \approx L \cdot (2N^2 d + 4Nd^2 + 3N d m),$$6 (Jin et al., 14 Dec 2025).

Taken together, these works show that “position-reweighted” can refer not only to score multipliers but also to the preservation of sequence coordinates, native order, and adjacency constraints. In grounding, OCR, and video, those structural choices can be as consequential as the importance metric itself.

6. Empirical patterns, alternative views, and open directions

Across the literature, position-aware methods generally improve the accuracy-efficiency trade-off, especially under severe compression. GridPrune reports a 2.14× speedup on LLaVA-1.5-7B at 192 tokens and a 5.09× speedup on LLaVA-NeXT-7B at 320 tokens, with negligible overhead from scoring and grid allocation (Duan et al., 13 Nov 2025). IVC-Prune reports relative averages of 100.6% on Qwen2.5-VL-7B, 99.6% on InternVL 2.5-8B, 100.1% on DeepSeek-VL2-16B, and 101.3% on LLaVA-v1.5-7B on general benchmarks while retaining about 50% of visual tokens (Sun et al., 3 Feb 2026).

At the same time, the literature does not converge on a single explanation of where position should enter. LearnPruner argues that vision encoders suffer from attention sink, while text-to-vision attention in middle LLM layers demonstrates resistance to positional bias and can provide effective pruning guidance; this is an alternative to explicit position correction, because it treats modality choice and layer choice as the main remedy (Takezoe et al., 27 Apr 2026). VisionDrop makes the opposite case against text-guided scoring, arguing that causal, semantic, and spatial forms of cross-modal misalignment undermine text-guided visual token reduction and showing that visual-only scoring can outperform text-guided scoring under matched budgets (Xu et al., 27 Jun 2025). HAWK sidesteps the issue differently: it omits RoPE when computing text-guided attention for scoring, explicitly to eliminate positional bias, and then reweights tokens by head importance rather than by raw attention alone (Zhu et al., 9 Apr 2026).

A further qualification comes from layer-depth analyses. “All You Need Are Random Visual Tokens?” reports that in deeper layers existing training-free pruning methods perform no better than random pruning, introduces an “information horizon” beyond which visual tokens become redundant, and shows that this horizon moves deeper for visually intensive tasks and for stronger models such as Qwen2.5-VL (Wang et al., 8 Dec 2025). This suggests that explicit position reweighting is most consequential before visual information has become uniform across tokens.

The overall trajectory of the field suggests a convergence toward hybrid designs. Existing papers already point to adaptive zones, 3D spatio-temporal grids, native-order preservation, anchor-first selection, and layer-wise routing as natural extensions (Duan et al., 13 Nov 2025, Jin et al., 14 Dec 2025, Chen et al., 12 Jun 2026). A plausible implication is that future position-reweighted pruning will combine at least three elements: a content score, an explicitly spatial budget or constraint, and a structural guarantee that preserved tokens continue to occupy meaningful positions in the model’s internal coordinate system.