Adaptive Token Merging

• Adaptive token merging is a dynamic Transformer sequence-reduction method that combines similar tokens based on data redundancy, structure, and layer depth.
• It employs various strategies such as threshold-based merging, depth-aware schedules, and graph-based matching to balance accuracy with efficiency in models like ViTs and diffusion systems.
• Empirical results show significant FLOPs reduction and speedup with minimal accuracy loss, though challenges remain in preserving structure and ensuring generative compatibility.

Adaptive token merging is a family of Transformer sequence-reduction methods in which tokens are combined according to data-dependent redundancy, layer depth, structural locality, or deployment constraints, rather than being reduced by a fixed pooling rule or discarded outright. A canonical starting point is Token Merging (ToMe), which gradually combines similar tokens inside a Vision Transformer without retraining (Bolya et al., 2022). Subsequent work extended the idea into threshold-based merging, depth-aware prune-then-merge schedules, multimodal pre-LLM compression, reversible dense-prediction variants, and runtime policies optimized jointly for accuracy, FLOPs, latency, and communication cost (Kim et al., 2023, Norouzi et al., 2024, Zhong et al., 2024, Erak et al., 12 Sep 2025). In contemporary usage, “adaptive” therefore refers not to one mechanism but to a design space: which tokens are merged, how many are merged, where merging occurs in depth, whether structure must be preserved, and whether the merge policy itself changes with the input or operating regime.

1. Conceptual basis and historical development

The basic rationale for token merging is that Transformer token sequences contain substantial redundancy. In ToMe, many ViT tokens become similar as they move through the network, so the model can combine redundant tokens rather than prune them; the method is applied inside each transformer block, between attention and MLP, and typically merges a fixed number $r$ of tokens per layer while choosing the pairs adaptively from the current features (Bolya et al., 2022). This distinction is important: the merge budget may be fixed, but the merge assignment is input-dependent.

Later methods broadened both the meaning and the scope of adaptivity. ToFu argues that token pruning is preferable when the model is sensitive to input interpolation, while token merging is preferable when the network response is approximately linear; it operationalizes this with a depth-based schedule in which early layers use pruned merging and later layers use average merging or MLERP merging (Kim et al., 2023). PiToMe modifies the original bipartite soft matching pipeline by estimating an energy score on a token graph, protecting informative low-energy tokens, and merging only high-energy candidates, with the stated goal of preserving the intrinsic spectral properties of the original token space (Tran et al., 2024). ATM makes the number of merges vary across layers and batches by using layer-specific similarity thresholds rather than a fixed top-$r$ rule, explicitly targeting “lossless token merging even without fine-tuning” on pretrained ViTs (Lee et al., 21 May 2025).

Multimodal systems added another axis of variation: merging can occur before a LLM, within it, or both. AIM places iterative token merging before the multimodal LLM, on visual tokens only, and then prunes visual tokens progressively inside the LLM based on multimodal importance (Zhong et al., 2024). This separation between pre-LLM compression and in-LLM reduction is conceptually distinct from earlier ViT-only methods, because it treats visual redundancy and reasoning dynamics as different compression problems.

2. Core algorithmic mechanisms

Despite their diversity, most adaptive token merging methods share a common computational pattern. Tokens are partitioned into candidate source and destination sets, pairwise similarity is computed, each source token is matched to a destination token, and selected pairs are fused into a shorter sequence. In ToMe, the partition is typically alternating, similarity is computed from normalized self-attention keys, and the $r$ strongest matches are merged by weighted averaging while tracking token sizes (Bolya et al., 2022). This bipartite soft matching design became the reference template for many later methods.

A second pattern is threshold-controlled merging. ALGM does not merge a predetermined number of tokens; instead, it uses a similarity threshold derived from training-set statistics, applies Conditional Local Average Pooling in the first layer within small local windows, and later applies Global Bipartite Matching across the entire image at an intermediate layer (Norouzi et al., 2024). ATM similarly replaces a fixed top-$r$ policy with layer-dependent similarity thresholds, using stricter thresholds in early layers and looser ones later, and further introduces size-distinctive matching so that large merged tokens are not merged with each other in final layers (Lee et al., 21 May 2025).

The merge operator itself is not uniform across the literature. Average merging is the simplest form, but several papers identify its failure modes. ToFu emphasizes that average merging changes the norm of the representation and can induce a distributional shift; MLERP is introduced as a norm-preserving alternative tailored to merge multiple tokens while maintaining the geometry and norm distribution more faithfully than averaging (Kim et al., 2023). In semantic communication, a norm-weighted merge gives higher influence to tokens with larger norms rather than performing an unweighted average (Erak et al., 11 Sep 2025). CubistMerge uses a different rule entirely: for each feature dimension, the merged token retains the value with the largest absolute magnitude among the merged tokens, which is intended to preserve salient activations without attention rescaling (Gong et al., 26 Sep 2025).

A fourth recurring mechanism is reversibility. In diffusion and segmentation, dense output requires every spatial position to remain addressable. ToMe for Stable Diffusion therefore merges tokens temporarily inside U-Net transformer blocks and then unmerges by duplicating the merged representation back to original positions (Bolya et al., 2023). ALGM stores merge indices, processes the reduced set, and restores the original token grid before segmentation output (Norouzi et al., 2024). In these settings, adaptive token merging is not a permanent compression of the representation space but a reversible compute-saving transformation.

3. Forms of adaptivity

Adaptive token merging has acquired several technically distinct meanings.

Adaptation mode	Representative mechanism	Example papers
Input-dependent pair selection	Similarity-based bipartite matching with fixed merge budget	ToMe (Bolya et al., 2022)
Threshold-based variable merging	Merge only when similarity exceeds a learned or configured threshold	ALGM (Norouzi et al., 2024), ATM (Lee et al., 21 May 2025)
Depth-dependent policy	Early pruning or conservative merging, later aggressive merging	ToFu (Kim et al., 2023), RAPID (Choi et al., 6 Jun 2026)
Structure-constrained locality	Within-frame, local-window, or path-graph merging only	AIM (Zhong et al., 2024), CubistMerge (Gong et al., 26 Sep 2025), local merging for time series (Götz et al., 2024)
System-level policy search	Per-layer ratios or thresholds optimized for Pareto trade-offs	(Erak et al., 11 Sep 2025, Erak et al., 12 Sep 2025)

One common misconception is that adaptive token merging always implies a variable token count per sample. ToMe is adaptive in the choice of merge pairs but typically uses a fixed merge budget $r$ per layer, precisely to preserve batching efficiency (Bolya et al., 2022). By contrast, ALGM and ATM make the number of remaining tokens depend on similarity thresholds, so simpler images can be merged more aggressively and more complex images can retain more tokens (Norouzi et al., 2024, Lee et al., 21 May 2025).

Another axis is depth. ToFu implements adaptation by representation regime: pruning is favored in early layers, where feature spaces are less linearly stable, and merging is favored later, where averaging-style interpolation becomes more appropriate (Kim et al., 2023). RAPID uses a related but not identical rationale: shallow-to-middle layers are handled by redundancy-aware pruning, whereas middle-to-late layers use importance-aware merging based on cosine similarity and CLS-token attention, with the reported best switch point occurring after the 4th layer in the tested configuration (Choi et al., 6 Jun 2026).

A third axis is deployment-aware optimization. In semantic communication, one paper formulates per-layer merge ratios $p_l$ as a multi-objective optimization over task accuracy and FLOPs, using Gaussian process Bayesian optimization to construct a Pareto frontier (Erak et al., 11 Sep 2025). A related edge semantic communication framework instead optimizes per-layer similarity thresholds $\tau_l$ against accuracy, compute, and communication cost, again with Bayesian optimization and Expected Hypervolume Improvement (Erak et al., 12 Sep 2025). In these cases, adaptivity is not only data-dependent within a forward pass; it is also policy-dependent across operating points.

4. Architectural variants and application domains

Adaptive token merging is no longer confined to image classification. In ViT classification, the dominant line runs from ToMe through ToFu, PiToMe, RAPID, and ATM, each modifying the pair-selection rule or the layer schedule while remaining training-free or near training-free on pretrained backbones (Bolya et al., 2022, Kim et al., 2023, Tran et al., 2024, Choi et al., 6 Jun 2026, Lee et al., 21 May 2025). CubistMerge extends this trajectory to spatial backbones such as SAM, MViTv2, ViTDet, SAM2/Hiera, Swin-based Mask2Former, and DINOv3 by enforcing a structured 2D reduction that preserves the spatial layout required by window attention, relative positional schemes, and RoPE (Gong et al., 26 Sep 2025).

Dense prediction introduced stricter structural constraints. ALGM is designed for semantic segmentation and merges twice—local first, global later—while explicitly restoring the token grid before prediction (Norouzi et al., 2024). The paper argues that this is a major difference from pruning methods, which permanently discard tokens and are harder to use for segmentation.

Diffusion models impose a different constraint: generation must preserve every spatial location through repeated denoising steps. ToMe for Stable Diffusion therefore applies merging selectively inside self-attention and only in high-token-count blocks, with unmerging afterward (Bolya et al., 2023). CA-ToMe adds an adaptive threshold and a cache of similar pairs across adjacent denoising steps, motivated by the observation that token relationships change only gradually over diffusion time (Saghatchian et al., 1 Jan 2025). In variable-length diffusion tokenization, however, data-dependent merging becomes problematic because the merge pattern is unavailable before generation; learnable global merging solves this by making the merge pattern data-independent and known in advance for each token length (Lee et al., 18 Jun 2026).

Multimodal LLM systems use adaptive token merging primarily to relieve the visual-token bottleneck. AIM performs iterative token merging on visual tokens before they enter the LLM and then prunes visual tokens progressively within the LLM, while keeping all text tokens (Zhong et al., 2024). DyTo applies hierarchical frame selection followed by dynamic intra-frame bipartite token merging for zero-shot video understanding (Zhang et al., 2024). TEAM-VLA uses similarity-based expansion before the language backbone and action-aware soft bipartite merging in a deeper layer of a vision-language-action model, making compression depend jointly on the observation and the task (Ye et al., 10 Dec 2025).

The same principle has also been adapted to non-visual sequences. For time series, local merging restricts candidate merges to nearby tokens, which lets the complexity of the similarity computation interpolate from linear to quadratic according to the neighborhood size $k$, and enables causal merging in transformer decoders (Götz et al., 2024). In genomics, MergeDNA uses differentiable token merging blocks with local-window constraints to learn a dynamic tokenizer that chunks adjacent bases into variable-length units, coupled to Merged Token Reconstruction and Adaptive Masked Token Modeling (Li et al., 17 Nov 2025).

5. Empirical behavior and reported trade-offs

The reported empirical range is broad, but the literature is consistent on one point: large fractions of tokens can often be merged before task performance degrades sharply.

ToMe reports up to $2\times$ throughput on ViT-L @ 512 and ViT-H @ 518 on images, and $2.2\times$ throughput on ViT-L on video, with only a $r$0 accuracy drop in each case (Bolya et al., 2022). PiToMe reports saving from $r$1 FLOPs of the base models, with a $r$2 average performance drop of ViT-MAE-H compared to $r$3 for baselines, and a $r$4 average performance drop of CLIP on Flickr30k compared to $r$5 as others (Tran et al., 2024). ATM reports over a $r$6 reduction in FLOPs for DeiT-T and DeiT-S without any drop in original accuracy, including $r$7 FLOPs reduction for DeiT-T and $r$8 for DeiT-S in the reported stress test (Lee et al., 21 May 2025).

In multimodal LLMs, AIM provides a particularly explicit ablation of merge ratio. On VideoMME, lowering the retention ratio from $r$9 to $r$0 reduces FLOPs from $r$1 TB to $r$2 TB while keeping performance nearly unchanged at $r$3 versus $r$4 baseline accuracy; for the full system, FLOPs drop to $r$5 TB versus $r$6 TB for the base LLaVA-OV-7B model, a $r$7 reduction (Zhong et al., 2024). The same paper reports a $r$8-fold reduction in FLOPs overall and a $r$9 improvement on MLVU under a similar computational cost by enabling denser temporal sampling.

For segmentation, ALGM reports up to $r$0 throughput improvement and up to $r$1 mIoU improvement. On ADE20K with Segmenter-S, the reported change is $r$2 mIoU and $r$3 im/sec, while GFLOPs drop from $r$4 to $r$5 (Norouzi et al., 2024). On Segmenter-L, ALGM* increases throughput from $r$6 to $r$7 im/sec while maintaining mIoU at $r$8.

For diffusion, ToMe for Stable Diffusion reports up to $r$9 token reduction, about $r$0 speedup, and up to $r$1 less memory with little to no loss in image quality (Bolya et al., 2023). CA-ToMe reports a speedup factor of $r$2 in the denoising process while maintaining the same FID scores compared to existing approaches, with the final table showing baseline $r$3 s and FID $r$4, ToMe $r$5 s and FID $r$6, and CA-ToMe $r$7 s and FID $r$8 (Saghatchian et al., 1 Jan 2025).

Outside vision, local merging for time series reports up to $r$9 acceleration on Chronos, with a reported $p_l$0 inference speedup in the fastest setting and a $p_l$1 speedup with a $p_l$2 MSE improvement on ETTh1 in the best-accuracy setting (Götz et al., 2024). These results suggest that the redundancy exploited by adaptive token merging is not specific to image patches.

6. Limitations, misconceptions, and open technical issues

A recurring misconception is that merging is uniformly preferable to pruning. ToFu explicitly rejects that view: pruning is preferable when the model is sensitive to input interpolation or when features are not yet well-behaved under linear combination, while merging is preferable when the network response is approximately linear around the tokens being combined (Kim et al., 2023). RAPID reaches a similar conclusion through a different criterion, switching from redundancy-aware pruning to importance-aware merging as semantic structure emerges with depth (Choi et al., 6 Jun 2026). Adaptive token merging is therefore often best understood as one component in a broader reduction policy, not as a universal replacement for pruning.

Preserving structure is another unresolved constraint. AIM reports that video merging should be done within each frame, not across frames, because cross-frame merging can hurt performance by disrupting temporal order and removing information important for long-video understanding (Zhong et al., 2024). CubistMerge argues that global merging strategies break the 2D organization required by window attention, decomposed relative positional embeddings, and RoPE, and therefore redesigns merging around path-graph locality and structured 2D reduction (Gong et al., 26 Sep 2025). Time-series local merging makes the same point for ordered sequences, where unrestricted all-to-all merging can violate temporal inductive bias (Götz et al., 2024).

A second limitation concerns representation geometry. Average merging is computationally simple, but ToFu shows that it can change the norm of the representation and induce distributional shift, motivating MLERP (Kim et al., 2023). CubistMerge replaces averaging with max-magnitude-per-dimension fusion for related reasons (Gong et al., 26 Sep 2025). This suggests that merge-pair selection and merge-operator design are separable problems, and that accuracy loss may arise from either one.

A third issue is generative compatibility. Data-dependent merging is attractive for recognition, but variable-length diffusion tokenization shows that generation requires the merge pattern to be known before the sample exists; learnable global merging is introduced precisely because conventional data-dependent merging makes the pattern inaccessible during generation (Lee et al., 18 Jun 2026). QuickMerge++ reaches a related conclusion for autoregressive settings, adding a lightweight transformer prior so that merged sequences remain compatible with next-token prediction (Liu et al., 16 Aug 2025). A plausible implication is that the strongest adaptive policies for analysis models are not automatically usable in synthesis models.

Finally, several frameworks achieve adaptivity only through additional tuning machinery. Bayesian optimization is used to discover Pareto-optimal merge schedules in semantic communication (Erak et al., 11 Sep 2025), and threshold policies likewise require offline multi-objective search in an edge transformer setting (Erak et al., 12 Sep 2025). TEAM-VLA reports that the expansion threshold $p_l$3, kernel size $p_l$4, context ratio $p_l$5, and final source budget $p_l$6 still need task-specific tuning (Ye et al., 10 Dec 2025). This suggests that adaptive token merging has shifted the optimization burden rather than eliminated it: fixed compression ratios are replaced by threshold, locality, importance, and budget policies whose behavior is more flexible but also more configuration-sensitive.