TransPrune: Inference-Time Token Pruning

• The paper introduces TransPrune, a training-free, progressive token pruning method for LVLMs that leverages token transition variation and instruction-guided attention.
• It computes token importance by measuring both magnitude and directional transition changes across transformer layers, avoiding attention’s positional bias.
• Empirical results on LLaVA models demonstrate that TransPrune cuts inference TFLOPs by over 50% while maintaining high accuracy across eight benchmarks.

TransPrune is a training-free, progressive inference-time token pruning method for large vision-LLMs (LVLMs) that estimates image-token importance from token transitions rather than relying only on attention. It was introduced for projector-based LVLMs such as LLaVA-v1.5-7B and LLaVA-NeXT-7B, where long visual token sequences dominate inference cost. The method combines Token Transition Variation (TTV)—a signal derived from changes in token representations across transformer modules—with Instruction-Guided Attention (IGA), then prunes image tokens at multiple layers while preserving multimodal performance across eight benchmarks and reducing inference TFLOPs by more than half (Li et al., 28 Jul 2025).

1. Conceptual basis and scope

TransPrune addresses the cost of carrying large numbers of visual tokens through the LLM. In the motivating setting, projector-based LVLMs such as LLaVA process image patch tokens inside the LLM; this burden becomes particularly large in high-resolution configurations, where LLaVA-v1.5 uses 576 image tokens and LLaVA-NeXT can use up to 2880 image tokens. Since transformer cost scales strongly with sequence length, pruning redundant visual tokens is a direct route to lower inference cost (Li et al., 28 Jul 2025).

The method is framed as an alternative to purely attention-based token scoring. The paper argues that attention-based criteria inherently suffer from limitations such as positional bias, and that attention may emphasize visually salient regions that are not the most semantically informative for the task. TransPrune therefore asks whether the transition of token representations across transformer modules provides a better signal of semantic importance. Its answer is affirmative: token transitions are treated as evidence of how actively the model is processing a token, while IGA supplies the instruction-conditioned component that TTV alone does not capture (Li et al., 28 Jul 2025).

This places TransPrune within the category of inference-time token pruning, not weight pruning, head pruning, or low-rank compression. The method is training-free, requires no calibration set, no auxiliary text encoder, and no learned predictor module, and is inserted during inference inside the LLM stack rather than in the vision encoder. Tokens are pruned/dropped, not merged, and the procedure is explicitly progressive across layers (Li et al., 28 Jul 2025).

2. Token Transition Variation

The core technical novelty is Token Transition Variation, which quantifies how much each image token changes when passing through a transformer module. If a token enters a module as $T_{\text{in}}$ and leaves as

$T_{\text{out}} = F(T_{\text{in}}),$

TransPrune measures two aspects of that change. The first is magnitude transition,

$$m(F, T_{\text{in}}) = \frac{\|T_{\text{out}}\|_2}{\|T_{\text{in}}\|_2},$$

and the second is direction transition,

$$d(F, T_{\text{in}}) = \frac{T_{\text{out}} \cdot T_{\text{in}}}{\|T_{\text{out}}\|_2 \, \|T_{\text{in}}\|_2}.$$

The paper’s empirical claim is that, in the shallow-to-middle layers it inspects, semantically important tokens often exhibit larger output/input norm ratios and more orthogonal directional change, i.e. smaller absolute cosine similarity between input and output. TTV combines these signals as

$$\text{TTV}(F, T_{I}) = \text{Softmax}\left(1 - \left| d(F, T_{I}) \right| \right) \cdot m(F, T_{I}),$$

where the softmax is applied across image tokens (Li et al., 28 Jul 2025).

At the layer level, TransPrune sums transition scores from both the self-attention and FFN submodules:

$$\text{TTV}_{l}(T_{I}) = \text{TTV}(Attention, T_{I}) + \text{TTV}(FFN, T_{I}).$$

Because single-layer transition patterns are not stable across all layers, TransPrune does not prune from a single layer’s score alone. Instead it accumulates TTV across a selected layer set $\mathcal{A}$. For a pruning layer $p_i$,

$$\text{TTV}_{p_i}(T_{I}) = \sum_{l \in \mathcal{A}, \, l \leq p_i} \text{TTV}_l(T_{I}).$$

In the reported implementation, TTV is accumulated across layers 7 to 12, and the paper’s layer-selection ablation shows that layers 7–12 outperform layers 1–6, supporting the interpretation that very early layers mostly encode low-level visual or positional information while later shallow-to-middle layers better reflect task-relevant semantics (Li et al., 28 Jul 2025).

3. Instruction-Guided Attention and progressive pruning

TTV is deliberately instruction-agnostic, so TransPrune adds Instruction-Guided Attention to reintroduce task relevance. Let $A_j$ denote the attention from the $T_{\text{out}} = F(T_{\text{in}}),$0-th instruction token to the image tokens, and let $T_{\text{out}} = F(T_{\text{in}}),$1 be the number of instruction tokens. Then

$T_{\text{out}} = F(T_{\text{in}}),$2

IGA measures how strongly the instruction attends to image tokens via attention inside the transformer. The final per-token pruning score at pruning layer $T_{\text{out}} = F(T_{\text{in}}),$3 is

$T_{\text{out}} = F(T_{\text{in}}),$4

with the paper using

$T_{\text{out}} = F(T_{\text{in}}),$5

Importantly, accumulation is applied only to TTV; IGA is not accumulated (Li et al., 28 Jul 2025).

TransPrune then ranks image tokens by this fused score and prunes those with lower values. The paper instantiates a 3-stage progressive schedule with pruning at layers 7, 9, and 12. Two operating points are reported. TransPrune-High uses retained ratios $T_{\text{out}} = F(T_{\text{in}}),$6 and ends with 72 image tokens. TransPrune-Low uses retained ratios $T_{\text{out}} = F(T_{\text{in}}),$7 and ends with 36 image tokens. These ratios are relative to the original number of image tokens (Li et al., 28 Jul 2025).

The design clarifies a common misconception. TransPrune is not simply “attention pruning with an extra heuristic.” Its central claim is that token self-transition provides a distinct, non-attention importance signal. The paper’s positional-bias analysis reinforces this distinction: IGA shows strong positional bias, preferring beginning and end positions, whereas TTV shows no clear positional bias and focuses more uniformly, especially near the image center (Li et al., 28 Jul 2025).

4. Computational characteristics and implementation

The method is designed so that the scoring overhead is small relative to the savings from shorter token sequences. With $T_{\text{out}} = F(T_{\text{in}}),$8 instruction tokens, hidden dimension $T_{\text{out}} = F(T_{\text{in}}),$9, FFN intermediate dimension $$m(F, T_{\text{in}}) = \frac{\|T_{\text{out}}\|_2}{\|T_{\text{in}}\|_2},$$0, $$m(F, T_{\text{in}}) = \frac{\|T_{\text{out}}\|_2}{\|T_{\text{in}}\|_2},$$1 pruning stages, $$m(F, T_{\text{in}}) = \frac{\|T_{\text{out}}\|_2}{\|T_{\text{in}}\|_2},$$2 retained visual tokens at stage $$m(F, T_{\text{in}}) = \frac{\|T_{\text{out}}\|_2}{\|T_{\text{in}}\|_2},$$3, and $$m(F, T_{\text{in}}) = \frac{\|T_{\text{out}}\|_2}{\|T_{\text{in}}\|_2},$$4 layers in stage $$m(F, T_{\text{in}}) = \frac{\|T_{\text{out}}\|_2}{\|T_{\text{in}}\|_2},$$5, the paper approximates total FLOPs as

$$m(F, T_{\text{in}}) = \frac{\|T_{\text{out}}\|_2}{\|T_{\text{in}}\|_2},$$6

The first term is transformer computation on retained tokens, the second term is instruction-to-image attention for IGA, and the last term is the TTV overhead. The paper states that, compared with the baseline model, the extra cost introduced by TransPrune is marginal (Li et al., 28 Jul 2025).

A practical implementation detail is that TransPrune computes only the needed attention weights from instruction tokens to image tokens, which helps preserve compatibility with FlashAttention. TTV itself does not require attention maps beyond module input/output representations, and the paper emphasizes that TTV alone does not require attention at all. This is one reason the method remains lightweight and training-free (Li et al., 28 Jul 2025).

The insertion point is also important. TransPrune is applied inside the LLM, after visual tokens have already been projected, rather than in the image encoder. This differs from projector-side compression methods such as VisionZip, but the paper demonstrates that the two are compatible. On LLaVA-v1.5-7B, at the retained 36 tokens setting, VisionZip alone uses 1.89 TFLOPs (49.5%), whereas VisionZip + TransPrune uses 0.66 TFLOPs (17.3%) with 98.0 accuracy. At the retained 24 tokens setting, VisionZip alone uses 1.25 TFLOPs (32.7%), whereas VisionZip + TransPrune uses 0.44 TFLOPs (11.5%) with 97.2 accuracy (Li et al., 28 Jul 2025).

5. Empirical results

The paper evaluates TransPrune on MME, MMBench, SEED, ScienceQA, VQA-v2, POPE, GQA, and TextVQA, using LLaVA-v1.5-7B and LLaVA-NeXT-7B as backbones and comparing against FastV, PDrop, SparseVLM, TopV, and ShortV (Li et al., 28 Jul 2025).

The main reported operating points are summarized below.

Model	Setting	TFLOPs / relative cost / average accuracy
LLaVA-v1.5-7B	Baseline	3.82 / 100% / 100.0
LLaVA-v1.5-7B	TransPrune-High	1.56 / 40.8% / 100.0
LLaVA-v1.5-7B	TransPrune-Low	1.19 / 31.2% / 98.4
LLaVA-NeXT-7B	Baseline	20.83 / 100% / 100.0
LLaVA-NeXT-7B	TransPrune-High	8.33 / 40.0% / 99.8
LLaVA-NeXT-7B	TransPrune-Low	6.41 / 30.8% / 98.4

On LLaVA-v1.5-7B, TransPrune-High preserves the paper’s normalized average accuracy at 100.0 while reducing cost to 1.56 TFLOPs (40.8%). On LLaVA-NeXT-7B, TransPrune-High achieves 8.33 TFLOPs (40.0%) with 99.8 average accuracy. These numbers underpin the paper’s central claim that TransPrune can maintain comparable multimodal performance while reducing inference TFLOPs by more than half (Li et al., 28 Jul 2025).

The ablations clarify the roles of individual components. A TTV-only version reaches 1.56 TFLOPs (40.8%) with 95.5 accuracy, compared with FastV at 2.01 TFLOPs (52.6%) and 97.8 accuracy. The paper interprets this as evidence that TTV alone can serve as an effective criterion without relying on attention, but also notes a significant drop on TextVQA, likely because TTV ignores instruction semantics. A complementary ablation shows that IGA + Magnitude outperforms IGA + Direction, while IGA + TTV is best overall, indicating that both transition magnitude and direction matter, with magnitude contributing more strongly (Li et al., 28 Jul 2025).

The accumulation mechanism also matters empirically. Without accumulation, the reported scores are lower or unchanged relative to the accumulated version; for example, MME rises from 1530 without accumulation to 1540 with accumulation, and MMBench rises from 65.7 to 66.0. The paper therefore treats accumulated transition history as a stabilization device rather than a cosmetic detail (Li et al., 28 Jul 2025).

6. Relation to prior token pruning and limitations

TransPrune is best understood as a specialized LVLM token-pruning method rather than a generic transformer pruning framework. Its novelty lies in replacing an attention-only importance estimate with a transition-based semantic signal, while still retaining instruction conditioning through IGA. The paper explicitly positions this against methods whose token scoring is dominated by attention, prompt relevance, or diversity heuristics, and it highlights positional bias as a key weakness of attention-only pruning (Li et al., 28 Jul 2025).

Several limitations are also explicit. First, TTV alone underperforms on instruction-dependent tasks such as TextVQA, which is why the full method fuses TTV with IGA. Second, token-transition patterns are not stable across all layers, which motivates the restriction to layers 7–12 and the choice of early pruning layers 7, 9, and 12. Third, the paper does not pursue pruning in very deep layers, stating that it does not use deeper layers beyond layer 20 because the goal is inference acceleration and early pruning yields better speedup. Fourth, the method is demonstrated on projector-based LVLMs—specifically LLaVA-v1.5-7B and LLaVA-NeXT-7B—so the scope of direct evidence is bounded by that model family (Li et al., 28 Jul 2025).

A final clarification concerns nomenclature. TransPrune is not a weight-pruning or architectural-pruning method in the sense used for LLM FFN sparsification or head pruning. It is a training-free, progressive, inference-time image-token pruning method whose distinguishing signal is Token Transition Variation. Its most general contribution is the claim that, in LVLMs, how a token changes across transformer modules can be a more reliable proxy for semantic importance than attention alone, especially when attention is affected by positional bias (Li et al., 28 Jul 2025).