---
title: Adaptive Token Merging
url: https://www.emergentmind.com/topics/adaptive-token-merging
type: topic
---

# Adaptive Token Merging

Adaptive token merging is a family of Transformer sequence-reduction methods in which tokens are combined according to data-dependent redundancy, layer depth, structural locality, or deployment constraints, rather than being reduced by a fixed pooling rule or discarded outright. A canonical starting point is Token Merging (ToMe), which gradually combines similar tokens inside a Vision Transformer without retraining [2210.09461]. Subsequent work extended the idea into threshold-based merging, depth-aware prune-then-merge schedules, multimodal pre-LLM compression, reversible dense-prediction variants, and runtime policies optimized jointly for accuracy, FLOPs, latency, and communication cost [2312.01026], [2406.09936], [2412.03248], [2509.09955]. In contemporary usage, “adaptive” therefore refers not to one mechanism but to a design space: which tokens are merged, how many are merged, where merging occurs in depth, whether structure must be preserved, and whether the merge policy itself changes with the input or operating regime.

## 1. Conceptual basis and historical development

The basic rationale for token merging is that Transformer token sequences contain substantial redundancy. In ToMe, many ViT tokens become similar as they move through the network, so the model can combine redundant tokens rather than prune them; the method is applied inside each transformer block, between attention and MLP, and typically merges a fixed number \(r\) of tokens per layer while choosing the pairs adaptively from the current features [2210.09461]. This distinction is important: the merge *budget* may be fixed, but the merge *assignment* is input-dependent.

Later methods broadened both the meaning and the scope of adaptivity. ToFu argues that token pruning is preferable when the model is sensitive to input interpolation, while token merging is preferable when the network response is approximately linear; it operationalizes this with a depth-based schedule in which early layers use pruned merging and later layers use average merging or MLERP merging [2312.01026]. PiToMe modifies the original bipartite soft matching pipeline by estimating an energy score on a token graph, protecting informative low-energy tokens, and merging only high-energy candidates, with the stated goal of preserving the intrinsic spectral properties of the original token space [2405.16148]. ATM makes the number of merges vary across layers and batches by using layer-specific similarity thresholds rather than a fixed top-\(r\) rule, explicitly targeting “lossless token merging even without fine-tuning” on pretrained ViTs [2505.15160].

Multimodal systems added another axis of variation: merging can occur before a large language model, within it, or both. AIM places iterative token merging before the multimodal LLM, on visual tokens only, and then prunes visual tokens progressively inside the LLM based on multimodal importance [2412.03248]. This separation between pre-LLM compression and in-LLM reduction is conceptually distinct from earlier ViT-only methods, because it treats visual redundancy and reasoning dynamics as different compression problems.

## 2. Core algorithmic mechanisms

Despite their diversity, most adaptive token merging methods share a common computational pattern. Tokens are partitioned into candidate source and destination sets, pairwise similarity is computed, each source token is matched to a destination token, and selected pairs are fused into a shorter sequence. In ToMe, the partition is typically alternating, similarity is computed from normalized self-attention keys, and the \(r\) strongest matches are merged by weighted averaging while tracking token sizes [2210.09461]. This bipartite soft matching design became the reference template for many later methods.

A second pattern is threshold-controlled merging. ALGM does not merge a predetermined number of tokens; instead, it uses a similarity threshold derived from training-set statistics, applies Conditional Local Average Pooling in the first layer within small local windows, and later applies Global Bipartite Matching across the entire image at an intermediate layer [2406.09936]. ATM similarly replaces a fixed top-\(r\) policy with layer-dependent similarity thresholds, using stricter thresholds in early layers and looser ones later, and further introduces size-distinctive matching so that large merged tokens are not merged with each other in final layers [2505.15160].

The merge operator itself is not uniform across the literature. Average merging is the simplest form, but several papers identify its failure modes. ToFu emphasizes that average merging changes the norm of the representation and can induce a distributional shift; MLERP is introduced as a norm-preserving alternative tailored to merge multiple tokens while maintaining the geometry and norm distribution more faithfully than averaging [2312.01026]. In semantic communication, a norm-weighted merge gives higher influence to tokens with larger norms rather than performing an unweighted average [2509.09168]. CubistMerge uses a different rule entirely: for each feature dimension, the merged token retains the value with the largest absolute magnitude among the merged tokens, which is intended to preserve salient activations without attention rescaling [2509.21764].

A fourth recurring mechanism is reversibility. In diffusion and segmentation, dense output requires every spatial position to remain addressable. ToMe for Stable Diffusion therefore merges tokens temporarily inside U-Net transformer blocks and then unmerges by duplicating the merged representation back to original positions [2303.17604]. ALGM stores merge indices, processes the reduced set, and restores the original token grid before segmentation output [2406.09936]. In these settings, adaptive token merging is not a permanent compression of the representation space but a reversible compute-saving transformation.

## 3. Forms of adaptivity

Adaptive token merging has acquired several technically distinct meanings.

| Adaptation mode | Representative mechanism | Example papers |
|---|---|---|
| Input-dependent pair selection | Similarity-based bipartite matching with fixed merge budget | ToMe [2210.09461] |
| Threshold-based variable merging | Merge only when similarity exceeds a learned or configured threshold | ALGM [2406.09936], ATM [2505.15160] |
| Depth-dependent policy | Early pruning or conservative merging, later aggressive merging | ToFu [2312.01026], RAPID [2606.08156] |
| Structure-constrained locality | Within-frame, local-window, or path-graph merging only | AIM [2412.03248], CubistMerge [2509.21764], local merging for time series [2405.17951] |
| System-level policy search | Per-layer ratios or thresholds optimized for Pareto trade-offs | [2509.09168], [2509.09955] |

One common misconception is that adaptive token merging always implies a variable token count per sample. ToMe is adaptive in the choice of merge pairs but typically uses a fixed merge budget \(r\) per layer, precisely to preserve batching efficiency [2210.09461]. By contrast, ALGM and ATM make the number of remaining tokens depend on similarity thresholds, so simpler images can be merged more aggressively and more complex images can retain more tokens [2406.09936], [2505.15160].

Another axis is depth. ToFu implements adaptation by representation regime: pruning is favored in early layers, where feature spaces are less linearly stable, and merging is favored later, where averaging-style interpolation becomes more appropriate [2312.01026]. RAPID uses a related but not identical rationale: shallow-to-middle layers are handled by redundancy-aware pruning, whereas middle-to-late layers use importance-aware merging based on cosine similarity and CLS-token attention, with the reported best switch point occurring after the 4th layer in the tested configuration [2606.08156].

A third axis is deployment-aware optimization. In semantic communication, one paper formulates per-layer merge ratios \(p_l\) as a multi-objective optimization over task accuracy and FLOPs, using Gaussian process Bayesian optimization to construct a Pareto frontier [2509.09168]. A related edge semantic communication framework instead optimizes per-layer similarity thresholds \(\tau_l\) against accuracy, compute, and communication cost, again with Bayesian optimization and Expected Hypervolume Improvement [2509.09955]. In these cases, adaptivity is not only data-dependent within a forward pass; it is also policy-dependent across operating points.

## 4. Architectural variants and application domains

Adaptive token merging is no longer confined to image classification. In ViT classification, the dominant line runs from ToMe through ToFu, PiToMe, RAPID, and ATM, each modifying the pair-selection rule or the layer schedule while remaining training-free or near training-free on pretrained backbones [2210.09461], [2312.01026], [2405.16148], [2606.08156], [2505.15160]. CubistMerge extends this trajectory to spatial backbones such as SAM, MViTv2, ViTDet, SAM2/Hiera, Swin-based Mask2Former, and DINOv3 by enforcing a structured 2D reduction that preserves the spatial layout required by window attention, relative positional schemes, and RoPE [2509.21764].

Dense prediction introduced stricter structural constraints. ALGM is designed for semantic segmentation and merges twice—local first, global later—while explicitly restoring the token grid before prediction [2406.09936]. The paper argues that this is a major difference from pruning methods, which permanently discard tokens and are harder to use for segmentation.

Diffusion models impose a different constraint: generation must preserve every spatial location through repeated denoising steps. ToMe for Stable Diffusion therefore applies merging selectively inside self-attention and only in high-token-count blocks, with unmerging afterward [2303.17604]. CA-ToMe adds an adaptive threshold and a cache of similar pairs across adjacent denoising steps, motivated by the observation that token relationships change only gradually over diffusion time [2501.00946]. In variable-length diffusion tokenization, however, data-dependent merging becomes problematic because the merge pattern is unavailable before generation; learnable global merging solves this by making the merge pattern data-independent and known in advance for each token length [2606.20076].

Multimodal LLM systems use adaptive token merging primarily to relieve the visual-token bottleneck. AIM performs iterative token merging on visual tokens before they enter the LLM and then prunes visual tokens progressively within the LLM, while keeping all text tokens [2412.03248]. DyTo applies hierarchical frame selection followed by dynamic intra-frame bipartite token merging for zero-shot video understanding [2411.14401]. TEAM-VLA uses similarity-based expansion before the language backbone and action-aware soft bipartite merging in a deeper layer of a vision-language-action model, making compression depend jointly on the observation and the task [2512.09927].

The same principle has also been adapted to non-visual sequences. For time series, local merging restricts candidate merges to nearby tokens, which lets the complexity of the similarity computation interpolate from linear to quadratic according to the neighborhood size \(k\), and enables causal merging in transformer decoders [2405.17951]. In genomics, MergeDNA uses differentiable token merging blocks with local-window constraints to learn a dynamic tokenizer that chunks adjacent bases into variable-length units, coupled to Merged Token Reconstruction and Adaptive Masked Token Modeling [2511.14806].

## 5. Empirical behavior and reported trade-offs

The reported empirical range is broad, but the literature is consistent on one point: large fractions of tokens can often be merged before task performance degrades sharply.

ToMe reports up to \(2\times\) throughput on ViT-L @ 512 and ViT-H @ 518 on images, and \(2.2\times\) throughput on ViT-L on video, with only a \(0.2\%-0.3\%\) accuracy drop in each case [2210.09461]. PiToMe reports saving from \(40\%-60\%\) FLOPs of the base models, with a \(0.5\%\) average performance drop of ViT-MAE-H compared to \(2.6\%\) for baselines, and a \(0.3\%\) average performance drop of CLIP on Flickr30k compared to \(4.5\%\) as others [2405.16148]. ATM reports over a \(30\%\) reduction in FLOPs for DeiT-T and DeiT-S without any drop in original accuracy, including \(31\%\) FLOPs reduction for DeiT-T and \(30\%\) for DeiT-S in the reported stress test [2505.15160].

In multimodal LLMs, AIM provides a particularly explicit ablation of merge ratio. On VideoMME, lowering the retention ratio from \(100\%\) to \(25\%\) reduces FLOPs from \(99.63\) TB to \(22.90\) TB while keeping performance nearly unchanged at \(58.0\) versus \(58.2\) baseline accuracy; for the full system, FLOPs drop to \(14.76\) TB versus \(99.63\) TB for the base LLaVA-OV-7B model, a \(6.8\times\) reduction [2412.03248]. The same paper reports a \(7\)-fold reduction in FLOPs overall and a \(+4.6\) improvement on MLVU under a similar computational cost by enabling denser temporal sampling.

For segmentation, ALGM reports up to \(100\%\) throughput improvement and up to \(+1.1\) mIoU improvement. On ADE20K with Segmenter-S, the reported change is \(45.3 \rightarrow 46.4\) mIoU and \(134 \rightarrow 192\) im/sec, while GFLOPs drop from \(38.6\) to \(26.3\) [2406.09936]. On Segmenter-L, ALGM\* increases throughput from \(10\) to \(20\) im/sec while maintaining mIoU at \(51.9\).

For diffusion, ToMe for Stable Diffusion reports up to \(60\%\) token reduction, about \(2\times\) speedup, and up to \(5.6\times\) less memory with little to no loss in image quality [2303.17604]. CA-ToMe reports a speedup factor of \(1.24\) in the denoising process while maintaining the same FID scores compared to existing approaches, with the final table showing baseline \(7.61\) s and FID \(33.66\), ToMe \(6.39\) s and FID \(34.16\), and CA-ToMe \(6.09\) s and FID \(34.05\) [2501.00946].

Outside vision, local merging for time series reports up to \(5400\%\) acceleration on Chronos, with a reported \(54.76\times\) inference speedup in the fastest setting and a \(14.17\times\) speedup with a \(6\%\) MSE improvement on ETTh1 in the best-accuracy setting [2405.17951]. These results suggest that the redundancy exploited by adaptive token merging is not specific to image patches.

## 6. Limitations, misconceptions, and open technical issues

A recurring misconception is that merging is uniformly preferable to pruning. ToFu explicitly rejects that view: pruning is preferable when the model is sensitive to input interpolation or when features are not yet well-behaved under linear combination, while merging is preferable when the network response is approximately linear around the tokens being combined [2312.01026]. RAPID reaches a similar conclusion through a different criterion, switching from redundancy-aware pruning to importance-aware merging as semantic structure emerges with depth [2606.08156]. Adaptive token merging is therefore often best understood as one component in a broader reduction policy, not as a universal replacement for pruning.

Preserving structure is another unresolved constraint. AIM reports that video merging should be done within each frame, not across frames, because cross-frame merging can hurt performance by disrupting temporal order and removing information important for long-video understanding [2412.03248]. CubistMerge argues that global merging strategies break the 2D organization required by window attention, decomposed relative positional embeddings, and RoPE, and therefore redesigns merging around path-graph locality and structured 2D reduction [2509.21764]. Time-series local merging makes the same point for ordered sequences, where unrestricted all-to-all merging can violate temporal inductive bias [2405.17951].

A second limitation concerns representation geometry. Average merging is computationally simple, but ToFu shows that it can change the norm of the representation and induce distributional shift, motivating MLERP [2312.01026]. CubistMerge replaces averaging with max-magnitude-per-dimension fusion for related reasons [2509.21764]. This suggests that merge-pair selection and merge-operator design are separable problems, and that accuracy loss may arise from either one.

A third issue is generative compatibility. Data-dependent merging is attractive for recognition, but variable-length diffusion tokenization shows that generation requires the merge pattern to be known before the sample exists; learnable global merging is introduced precisely because conventional data-dependent merging makes the pattern inaccessible during generation [2606.20076]. QuickMerge++ reaches a related conclusion for autoregressive settings, adding a lightweight transformer prior so that merged sequences remain compatible with next-token prediction [2508.13204]. A plausible implication is that the strongest adaptive policies for analysis models are not automatically usable in synthesis models.

Finally, several frameworks achieve adaptivity only through additional tuning machinery. Bayesian optimization is used to discover Pareto-optimal merge schedules in semantic communication [2509.09168], and threshold policies likewise require offline multi-objective search in an edge transformer setting [2509.09955]. TEAM-VLA reports that the expansion threshold \(\tau\), kernel size \(K\), context ratio \(u\), and final source budget \(M\) still need task-specific tuning [2512.09927]. This suggests that adaptive token merging has shifted the optimization burden rather than eliminated it: fixed compression ratios are replaced by threshold, locality, importance, and budget policies whose behavior is more flexible but also more configuration-sensitive.

Source: https://www.emergentmind.com/topics/adaptive-token-merging