Pyramid Token Merging in VLMs and ViTs

• Pyramid Token Merging is a hierarchical strategy that progressively merges low-importance tokens in vision–language models and vision transformers to reduce computational overhead.
• It employs attention-derived importance scores and weighted averaging to merge tokens, ensuring that critical visual information is preserved during token reduction.
• Empirical results demonstrate that PTM can maintain near-original model accuracy (up to 98%) while drastically lowering token counts and improving throughput.

Searching arXiv for the specified papers and closely related work on pyramid/token merging in VLMs and ViTs. Pyramid Token Merging (PTM) denotes a hierarchical token-reduction strategy in which low-importance tokens are merged progressively rather than discarded in a single step. In contemporary transformer systems, the motivation is the quadratic attention cost in sequence length, $O(N^2)$, together with the empirical redundancy of visual tokens. In the formulation introduced for vision–LLMs in LightVLM, PTM reduces image tokens across several LLM layers by finally only keeping a few dominant tokens, and it is described as training-free, with zero fine-tuning overhead, and compatible with Flash-Attention or vanilla dot-product attention (Hu et al., 30 Aug 2025). A related use of the same phrase appears in a ViT-efficiency setting, where a two-level pyramid of high-scale and low-scale image tokens is fused before token selection and mergence, again with the objective of reducing computation while preserving information from non-crucial tokens (Bian et al., 2023).

1. Conceptual scope and problem setting

In modern vision–LLMs, an image is first split into hundreds or thousands of patch tokens and then concatenated with text before being fed into an LLM. Because attention costs scale as $O(N^2)$, these many image tokens dominate both compute and memory, yet they exhibit high redundancy. LightVLM reports an empirical observation that in early LLM layers, attention to image tokens is roughly uniform, whereas in deep layers it concentrates on a small subset described as “attention sinks”; an example given is that in layer 27 fewer than 3% of image tokens cover $\sim 90\%$ of total attention weight (Hu et al., 30 Aug 2025).

In the ViT setting, the motivation is closely related but framed through token pruning. The central claim is that discarding tokens lowers computational overhead but also causes loss of potential information, so non-crucial tokens should instead be merged with more crucial tokens. The same work couples that idea to multi-scale feature construction, using a high-scale branch and a low-scale branch as a simple two-level pyramid before token mergence (Bian et al., 2023).

A useful distinction follows from these two formulations. In LightVLM, the “pyramid” is primarily a hierarchy of merge points distributed across LLM layers. In the ViT method, the “pyramid” refers first to multi-scale token generation and then to token reduction at designated transformer layers. This suggests that PTM is better understood as a family of hierarchical token-compression schemes than as a single fixed algorithm.

Formulation	Pyramid structure	Merge signal
LightVLM	$K$ merge points in the LLM stack	Per-token attention-weight sum averaged over heads
Multi-Scale And Token Mergence	Two patch resolutions plus merge modules at selected ViT layers	$[\mathrm{CLS}]$-to-token attention, then cosine similarity

2. Hierarchical merging in LightVLM

LightVLM organizes the LLM stack into $K$ merge points, for example layers $\{\ell_1,\ell_2,\ldots,\ell_K\}$. At each merge point $\ell_i$, it computes per-token importance scores $A \in \mathbb{R}^N$, either from the full row-sum of softmaxed $QK^\top/V$ weights or from the cumsum output of Flash-Attention, averaged over heads. The $O(N^2)$0 tokens are then partitioned into an unmerged set $O(N^2)$1 of size $O(N^2)$2, consisting of the top $O(N^2)$3 tokens ranked by $O(N^2)$4, and a to-be-merged set $O(N^2)$5 of size $O(N^2)$6, consisting of the bottom $O(N^2)$7 tokens. The tokens in $O(N^2)$8 are merged into a single super-token, and the merged token is concatenated with $O(N^2)$9, reducing the sequence length from $\sim 90\%$0 to $\sim 90\%$1 at that stage (Hu et al., 30 Aug 2025).

With input token matrix $\sim 90\%$2 and importance vector $\sim 90\%$3, the importance score for token $\sim 90\%$4 is given as

$\sim 90\%$5

If the bottom $\sim 90\%$6 tokens are collected in $\sim 90\%$7, the merged token can be formed by a weighted sum

$\sim 90\%$8

The weight vector is specified as a simple decreasing integer sequence $\sim 90\%$9 and may be normalized to sum to 1. This weighting emphasizes the more important tokens among the group (Hu et al., 30 Aug 2025).

The merge layers are described as three points, for example LLM layers 5, 9, and 13, intended to capture low-, mid-, and high-level features. The merge budgets $K$0 are chosen according to a target final token ratio $K$1, with

$K$2

Common presets are to keep 35%, 15%, or 3% of original image tokens. The intended effect is progressive reduction: only a few highly informative tokens remain in the deepest layers, while earlier layers still process a richer visual token set (Hu et al., 30 Aug 2025).

3. Multi-scale token mergence in ViTs

The ViT-oriented formulation begins with multi-scale feature generation. The input image is embedded at two patch resolutions, “high-scale” and “low-scale,” analogous to a two-level pyramid. In the cited example, the high-scale branch uses patch size $K$3 such as $K$4, grid size $K$5 such as $K$6, and hence $K$7, plus a $K$8 token, producing $K$9. The low-scale branch uses patch size $[\mathrm{CLS}]$0 such as $[\mathrm{CLS}]$1, grid size $[\mathrm{CLS}]$2 such as $[\mathrm{CLS}]$3, and hence $[\mathrm{CLS}]$4, again plus $[\mathrm{CLS}]$5, producing $[\mathrm{CLS}]$6. The two branches share the same embedding dimension $[\mathrm{CLS}]$7 but differ in token count (Bian et al., 2023).

Before token merging, the low-scale tokens are upsampled and fused into the high-scale stream:

$[\mathrm{CLS}]$8

Here, $[\mathrm{CLS}]$9 denotes nearest-neighbor upsampling from $K$0 to $K$1, LKA is a light-weight large-kernel attention module, and PEG re-applies positional encodings in token dimension. This fusion is performed once just after the patch-embedding stage (Bian et al., 2023).

At designated transformer layers, such as the 4th, 7th, and 10th layers in DeiT, token importance is computed from $K$2-to-token attention:

$K$3

After averaging over heads if multi-head, the tokens are sorted by descending $K$4. The top $K$5 are retained as crucial tokens, including $K$6, and the remaining $K$7 are treated as non-crucial. Each non-crucial token $K$8 is then assigned to the most similar crucial token $K$9 via cosine similarity,

$\{\ell_1,\ell_2,\ldots,\ell_K\}$0

If $\{\ell_1,\ell_2,\ldots,\ell_K\}$1 denotes the set of non-crucial tokens assigned to crucial token $\{\ell_1,\ell_2,\ldots,\ell_K\}$2, the merged token is

$\{\ell_1,\ell_2,\ldots,\ell_K\}$3

The next layer receives $\{\ell_1,\ell_2,\ldots,\ell_K\}$4 together with the merged tokens $\{\ell_1,\ell_2,\ldots,\ell_K\}$5 (Bian et al., 2023).

This construction differs from the LightVLM mechanism in one important respect. LightVLM merges the least-important tokens directly into a single super-token at each merge point. The ViT method first identifies crucial tokens and then re-aggregates each non-crucial token into its most similar crucial token. The shared principle is preservation of information through weighted merging rather than pure deletion.

4. Architectural integration and computational properties

In LightVLM, PTM is introduced as the encoding-stage acceleration component of a broader system that divides VLM inference into encoding and decoding. The stated requirement is seamless deployment upon existing VLMs, in a training-free manner. PTM is also described as fully compatible with Flash-Attention or vanilla dot-product attention because it only needs $\{\ell_1,\ell_2,\ldots,\ell_K\}$6, not the full $\{\ell_1,\ell_2,\ldots,\ell_K\}$7 attention map (Hu et al., 30 Aug 2025).

The computational effect is expressed through the reduction of the attention term from baseline per-layer cost $\{\ell_1,\ell_2,\ldots,\ell_K\}$8 to $\{\ell_1,\ell_2,\ldots,\ell_K\}$9 after one merge of $\ell_i$0 tokens. Over $\ell_i$1 merge points, the total cost is written roughly as

$\ell_i$2

Because $\ell_i$3, the deepest layers see quadratic benefits. Token-merging bookkeeping is described as sorting $\ell_i$4 of length $\ell_i$5, costing $\ell_i$6, though implementable via partial-select in $\ell_i$7, and argued to be negligible relative to attention $\ell_i$8. The overhead is summarized as $\ell_i$9 extra per merge step beyond slicing and a single small matrix multiply. Peak activations fall from $A \in \mathbb{R}^N$0 to $A \in \mathbb{R}^N$1 (Hu et al., 30 Aug 2025).

In the ViT method, multi-scale fusion introduces added work, but the architecture compensates by using a downsample–MHSA–upsample step in the first few high-scale blocks:

$A \in \mathbb{R}^N$2

This is stated to keep per-block FLOPs approximately original. Apart from multi-scale fusion and Token Selection + Mergence modules inserted into a small number of transformer layers, the standard MHSA+FFN blocks remain unchanged (Bian et al., 2023).

A common misconception is that hierarchical token reduction necessarily depends on retraining or learned routing. The sources considered here describe two different training-free constructions: one driven directly by LLM attention at merge points, and one based on $A \in \mathbb{R}^N$3 attention plus cosine-similarity assignment. Another misconception is that token reduction here is identical to token dropping. In both formulations, the defining operation is merging, which is intended to preserve information from tokens that would otherwise be removed.

5. Empirical results

For LightVLM, the reported empirical results are given for the combined system of PTM and KV-cache compression on QWen2.5-VL 7B and InternVL models. The system retains 100% of original performance when keeping 35% of image tokens and maintains around 98% performance when keeping only 3% of image tokens. It yields a $A \in \mathbb{R}^N$4 end-to-end network throughput gain on V100/A800 and a $A \in \mathbb{R}^N$5 reduction in prefilling time. For long decoding sequences of 4096 tokens, the reported gains are a $A \in \mathbb{R}^N$6 inference-time reduction and a $A \in \mathbb{R}^N$7 memory reduction. The same source also states that LightVLM enables a heavy model, exemplified by InternVL2.5 26B, to infer faster than significantly smaller models, exemplified by InternVL2.5 8B (Hu et al., 30 Aug 2025).

For the ViT method on ImageNet-1K, Table 1 reports the following comparisons. DeiT-S baseline is 22.1 M parameters, 4.6 G FLOPs, and 79.8% top-1; the modified model is 22.2 M, 3.1 G, and 79.7%, corresponding to a 32.6% FLOP reduction with a 0.1% accuracy decrease. DeiT-T changes from 5.7 M to 5.8 M and from 1.3 G to 0.8 G, while top-1 changes from 72.2% to 72.7%. DeiT-B changes from 17.5 G to 11.5 G FLOPs, with top-1 changing from 81.8% to 81.5%. Figure 1 is reported to show that at lower keep rates $A \in \mathbb{R}^N$8, the accuracy gap over DynamicViT and EViT widens up to approximately 2%. Table 3 gives an ablation on EViT-DeiT-S at $A \in \mathbb{R}^N$9: baseline 78.5%, +TM only 79.1%, and +TM+MF 79.2%. Table 2 reports cosine similarity at 79.2%, outperforming random, attention-map, $QK^\top/V$0, and $QK^\top/V$1 by approximately 0.1% (Bian et al., 2023).

These results jointly indicate that PTM-style methods are used in two distinct efficiency regimes. One regime targets VLM inference latency and throughput in the presence of long multimodal sequences. The other targets ViT FLOPs reduction while maintaining ImageNet accuracy. The common experimental pattern is that substantial token-count reduction can be tolerated when the retained or merged tokens are selected through attention-derived importance signals.

6. Relation to adjacent methods, trade-offs, and open directions

LightVLM is compared directly with several alternative token-reduction approaches. SparseVLM and SparseViT are characterized as learning a per-layer sparsity schedule via rank or learned thresholds and often being training-based. FasterVLM, described as a $QK^\top/V$2-prune method, drops tokens based on $QK^\top/V$3 patch attention and typically performs one-shot pruning after the vision encoder. PyramidDrop drops lowest-attention tokens at several layers, whereas PTM merges them and therefore preserves some information via weighted sum. VisionZip clusters patches by cross-patch attention and then prunes or merges, incurring extra clustering cost; PTM is described as using a single weighted sum per merge group, so overhead is approximately zero (Hu et al., 30 Aug 2025).

The ViT method is presented as orthogonal to many other token-pruning schemes and as a drop-in upgrade. Its stated trade-offs are that multi-scale fusion adds overhead, though the downsample–MHSA–upsample step recovers most of it; the greedy per-token assignment may not be optimal; the current construction uses a two-level pyramid; and the merging rule is a fixed weighted average rather than an attention-based or gating strategy. LightVLM lists related considerations of a different kind: merge points $QK^\top/V$4 and budgets $QK^\top/V$5 must be tuned to match compute or resource targets, and the weighting scheme $QK^\top/V$6 is heuristic, with mean or attention-weighted merges named as alternatives that could be tried (Bian et al., 2023, Hu et al., 30 Aug 2025).

A plausible implication is that “pyramid” should not be reduced to a single design template. In one line of work it denotes progressive token compression across depth; in another it combines a multi-scale token hierarchy with merge-based token selection. What remains consistent is the attempt to exploit attention-derived redundancy while avoiding the information loss associated with naive pruning.