Token Transition Variation in Vision-Language Models

• Token Transition Variation (TTV) is a metric that quantifies token importance by measuring changes in magnitude and direction of image token representations.
• It is applied during inference in LVLMs to prune tokens progressively, mitigating positional bias and overemphasis on visually salient but semantically weak regions.
• Combined with Instruction-Guided Attention in TransPrune, TTV delivers efficient model performance with reduced compute while maintaining competitive accuracy.

Token Transition Variation (TTV) is a token-importance metric introduced for inference-time pruning of visual tokens in large vision-LLMs (LVLMs). In TransPrune, TTV scores an image token by measuring how strongly its representation changes, both in magnitude and in direction, as it passes through the self-attention and FFN modules of the LLM. The method is explicitly training-free, operates in-LLM during inference, and is used either as a standalone criterion or jointly with Instruction-Guided Attention (IGA) to rank image tokens for progressive pruning. Its intended role is to provide a non-attention signal for token importance in settings where attention-based pruning is affected by positional bias or by overemphasis on visually salient but semantically weak regions (Li et al., 28 Jul 2025).

1. Definition and problem setting

TTV arises from a specific critique of existing token-pruning practice in LVLMs. Most prior in-LLM token pruning methods estimate importance primarily through attention-derived scores. In TransPrune, this is treated as an incomplete proxy: attention can exhibit positional bias, and it can highlight visually salient regions that are not necessarily the most semantically informative. TTV is proposed as an alternative perspective in which token importance is inferred from the token’s own representational evolution through the model, rather than from inter-token attention structure alone (Li et al., 28 Jul 2025).

The central empirical observation is that token transitions inside transformer layers carry semantic information. The authors report that, in shallow-to-mid layers, especially around layers 6–14, and more consistently for the accumulation design over layers 7–12, tokens corresponding to semantically important image regions often undergo larger changes in norm and larger directional changes. Magnitude change is measured through the ratio of output norm to input norm, while directional change is operationalized through lower absolute cosine similarity between input and output. This motivates TTV as a composite measure of representational amplification and reorientation.

Conceptually, TTV assigns high importance to tokens that the model actively transforms. A token with high TTV is one whose representation is both rescaled and reoriented by the model; a token with low TTV changes little in norm and/or direction and is therefore treated as relatively redundant or less informative. The method is defined only for image tokens $T_I$; instruction tokens are used elsewhere in TransPrune, but are not themselves the pruning target.

2. Mathematical formulation

TTV is defined at the level of a transformation module $F$, where $F$ is either the self-attention module or the FFN within a transformer layer. Let the module input be $T_{\text{in}}$, and let

$T_{\text{out}} = F(T_{\text{in}}).$

The transition is decomposed into a magnitude term and a direction term. The magnitude transition is

$$m(F, T_{\text{in}}) = \frac{\|T_{\text{out}}\|_2}{\|T_{\text{in}}\|_2},$$

which is the ratio of output to input $L_2$ norm. The direction transition is the cosine similarity

$$d(F, T_{\text{in}}) = \frac{T_{\text{out}} \cdot T_{\text{in}}}{\|T_{\text{out}}\|_2 \, \|T_{\text{in}}\|_2}.$$

Direction is then converted into a variation score by using $1 - |d(F,T_I)|$, so that more orthogonal input/output pairs receive larger values. For image tokens $T_I$, this term is normalized across tokens with softmax and multiplied by the magnitude term:

$F$0

This is the exact module-level definition of Token Transition Variation (Li et al., 28 Jul 2025).

At the layer level, TTV is computed separately for the attention and FFN modules and then added:

$F$1

An important methodological point is that TTV is not defined between consecutive residual-stream states of a full layer. It is computed within each module, using that module’s input and output representations.

Because the authors find that TTV behavior is not stable across all depths, they introduce accumulation over a selected layer set. If the accumulation layer set is

$F$2

and the pruning layer set is

$F$3

then for each pruning layer $F$4,

$F$5

In the experimental implementation, accumulation is performed over layers 7 to 12, while pruning occurs at layers 7, 9, and 12.

3. Use inside TransPrune

TransPrune is a training-free, progressive, in-LLM token-pruning method in which TTV is combined with Instruction-Guided Attention. The two signals are intended to be complementary: TTV measures intrinsic representational transformation of image tokens, while IGA measures instruction relevance (Li et al., 28 Jul 2025).

IGA is defined from attention between instruction-token queries and image-token keys. If $F$6 denotes the relevant attention matrix and $F$7 is the instruction length, then

$F$8

where $F$9 is the attention weight from the $F$0-th instruction token to image tokens.

At each pruning layer $F$1, the accumulated TTV score and IGA are combined as

$F$2

with $F$3. The experiments set $F$4. As stated in the paper, the indexing uses $F$5; the manuscript does not further elaborate on that indexing choice. Tokens with lower combined scores are pruned.

The pruning process is progressive. The model runs to a designated pruning layer, TTV is computed from the attention and FFN transitions of currently retained image tokens, the selected-layer TTV scores are accumulated, IGA is computed from instruction-to-image attention, the combined score is formed, and the top-ranked image tokens are retained according to a preset keep ratio. In the main configuration, accumulation layers are 7–12 and pruning layers are 7, 9, and 12. The keep ratios are $F$6 for TransPrune-High, yielding a final token count of 72, and $F$7 for TransPrune-Low, yielding a final token count of 36.

The method is presented as compatible with FlashAttention. The reason given is architectural rather than algorithmic novelty: TTV requires only module inputs and outputs, while IGA requires attention weights from instruction tokens to image tokens rather than the full attention map. The additional computational overhead attributed specifically to TTV is described as marginal, with asymptotic cost $F$8 in the paper’s FLOPs approximation.

4. Layer dependence and representational interpretation

The interpretive basis of TTV is explicitly layer-dependent. The paper reports that important tokens often show larger magnitude change and larger directional change in shallow-to-mid layers, especially around layers 6–14, and that the accumulation design is most effective over layers 7–12 (Li et al., 28 Jul 2025). This layer dependence is used as evidence against indiscriminate aggregation across the full depth of the model.

The magnitude term captures how strongly the model amplifies or suppresses a token representation. A large value of

$F$9

indicates strong rescaling. The direction term captures semantic transformation rather than mere preservation. Because TTV uses $T_{\text{in}}$0, a token receives a larger score when the output representation becomes less aligned with the input. Geometrically, the method privileges tokens whose representations become more orthogonal to their previous state while also being amplified.

The accumulation mechanism is introduced precisely because these transition patterns do not remain equally informative at all depths. The paper compares accumulated TTV from layers 1–6 against layers 7–12 while keeping pruning layers fixed at 7, 9, and 12. The later range performs better: MME improves from 1515 to 1540, SQA$T_{\text{in}}$1 from 69.4 to 69.5, GQA from 61.3 to 61.4, and MMB$T_{\text{in}}$2 from 65.6 to 66.0. The stated interpretation is that the earliest layers primarily encode lower-level visual or positional information, whereas the selected deeper shallow/mid layers are more informative for pruning.

A further ablation isolates the components of TTV. Starting from Only IGA, adding direction raises MME from 1514 to 1521, adding magnitude raises it to 1532, and using the full TTV combination reaches 1540; SQA changes from 69.0 to 69.1, 69.4, and 69.5, respectively, while MMB$T_{\text{in}}$3 changes from 65.6 to 65.4, 65.7, and 66.0. This supports two conclusions stated in the paper: both components contribute, and magnitude contributes more strongly than direction.

5. Empirical behavior in TransPrune

A key experiment evaluates TTV as a standalone importance signal on LLaVA-1.5-7B. In that setting, the baseline model uses 3.82 TFLOPs with normalized accuracy 100.0, FastV$T_{\text{in}}$4 uses 2.01 TFLOPs (52.6%) with accuracy 97.8, and TransPrune with TTV only uses 1.56 TFLOPs (40.8%) with accuracy 95.5 (Li et al., 28 Jul 2025).

Method	TFLOPs	Accuracy
LLaVA-1.5-7B baseline	3.82 (100%)	100.0
FastV$T_{\text{in}}$5	2.01 (52.6%)	97.8
TransPrune (TTV-only)	1.56 (40.8%)	95.5
TransPrune-High	1.56 (40.8%)	100.0
TransPrune-Low	1.19 (31.2%)	98.4

The TTV-only configuration therefore serves as a non-attention pruning criterion with performance comparable to attention-based methods, although the paper notes a noticeable drop on TextVQA, where TTV-only reaches 50.9. This is attributed to the lack of instruction dependence in TTV alone.

The full TransPrune configuration combines TTV and IGA and performs better than TTV-only. On LLaVA-v1.5-7B, TransPrune-High uses 1.56 TFLOPs, or 40.8% of baseline, while preserving average performance across eight benchmarks at 100.0. The per-benchmark values reported are MME$T_{\text{in}}$6 1540, VQA-v2 77.9, Seed$T_{\text{in}}$7 65.7, TextVQA 57.8, SQA$T_{\text{in}}$8 69.5, POPE 85.0, GQA 61.4, and MMB$T_{\text{in}}$9 66.0. TransPrune-Low reduces compute further to 1.19 TFLOPs (31.2%) with accuracy 98.4 and final token count 36.

On LLaVA-NeXT-7B, the same pattern is reported: TransPrune-High uses 8.33 TFLOPs (40.0%) with accuracy 99.8, and TransPrune-Low uses 6.41 TFLOPs (30.8%) with accuracy 98.4. The paper presents these results as evidence that TTV remains effective when used in its intended combined framework.

Additional ablations support the stability of the scoring design. Accumulating TTV across layers improves MME$T_{\text{out}} = F(T_{\text{in}}).$0 from 1530 to 1540, SQA$T_{\text{out}} = F(T_{\text{in}}).$1 from 69.2 to 69.5, and MMB$T_{\text{out}} = F(T_{\text{in}}).$2 from 65.7 to 66.0, while leaving GQA unchanged at 61.4. Varying the mixing coefficient $T_{\text{out}} = F(T_{\text{in}}).$3 gives MMB values of 65.5 at $T_{\text{out}} = F(T_{\text{in}}).$4, 66.0 at $T_{\text{out}} = F(T_{\text{in}}).$5, and 65.9 at $T_{\text{out}} = F(T_{\text{in}}).$6, which the paper interprets as support for equal weighting between TTV and IGA.

6. Relation to attention-based scoring, limitations, and nomenclature

The principal distinction between TTV and attention-based token scoring is methodological. Attention-based methods infer importance from inter-token dependency structure, such as attention from the last token or from instruction tokens. TTV instead evaluates a token by its self-transition through a module: its input/output norm ratio and input/output directional change. The paper associates this with two practical consequences: reduced positional bias, because the score is not determined by attention patterns shaped by masking and attention dynamics, and reduced reliance on visually salient but semantically weak regions (Li et al., 28 Jul 2025).

The empirical support for the positional-bias claim comes from the position-frequency analysis on MME. There, IGA retains tokens disproportionately at the beginning and end of the sequence, whereas TTV shows no obvious positional bias and retains tokens more uniformly, especially toward the center of the image. This is presented as one reason TTV complements IGA rather than simply duplicating it.

The limitations are equally explicit. TTV alone does not encode instruction-conditioned relevance, which is why it underperforms on TextVQA relative to the full TTV+IGA configuration. The method is also intentionally layer-selective rather than universal across depth; the usefulness of token transition patterns changes by layer. For this reason, the paper does not present TTV as a complete replacement for instruction-aware signals in all settings, but as a strong non-attention criterion that is most effective when combined with IGA.

The paper also treats TTV as a plug-and-play component of broader efficient LVLM inference systems. Because it depends only on module inputs and outputs, it is presented as straightforward to integrate into existing pipelines and as compatible with projector-based pruning methods such as VisionZip. In the reported combined experiments, VisionZip+TransPrune reduces TFLOPs further while keeping performance close to VisionZip alone.

The acronym is overloaded in arXiv literature. In LVLM pruning, TTV denotes Token Transition Variation (Li et al., 28 Jul 2025). In an unrelated exoplanet literature, TTV commonly denotes transit timing variations, a dynamical observable based on deviations from a linear transit ephemeris (Lithwick et al., 2012). The two uses share only the acronym; the underlying objects, models, and inference problems are unrelated.