---
title: Token Merging in Transformers
url: https://www.emergentmind.com/topics/token-merging
type: topic
---

# Token Merging in Transformers

Token merging is a class of architectural and inference-time techniques for reducing the effective sequence length in transformer and state-space models, primarily to accelerate computation and decrease memory and energy consumption. By adaptively collapsing redundant or semantically similar tokens into fewer representative tokens, these methods can cut the quadratic complexity of attention and enable efficient, scalable application to vision, language, sequence, and multimodal tasks with minimal accuracy loss. Token merging has rapidly evolved beyond early heuristic approaches, now encompassing energy- and importance-aware algorithms, integration with quantization and clustering, local and global policies, and domain-specific variants for dense prediction, time series, and code.

## 1. Mathematical Principles and Core Algorithms

Token merging operates by combining subsets of tokens in sequence models into single “super-tokens,” reducing sequence length and hence the cost of subsequent layers. The core components across most approaches are:

- **Similarity Computation:** Tokens are mapped to an embedding or key space, and similarity is measured via cosine similarity, dot product, or other learned metrics. For example, ToMe merges tokens $(i, j)$ using $S_{ij} = \cos(k_i, k_j)$ where $k_i$ are normalized key vectors [2210.09461].
- **Matching Strategy:** Pairs or groups of similar tokens are determined via matching algorithms. Classic bipartite soft matching partitions tokens into two groups (e.g., “even” and “odd”) and merges only across groups, while more advanced approaches use hierarchical agglomerative clustering with single, complete, or average linkage to select merge pairs or clusters [2409.11923].
- **Token Merge Operator:** Fused tokens are computed as weighted averages, with options for norm-preserving spherical interpolation (e.g., MLERP in ToFu) or component-wise fusion (e.g., max-per-dimension in CubistMerge) [2312.01026] [2509.21764].
- **Downstream Attention Update:** Merged tokens often represent the union of several patches or positions. To account for their increased “mass,” attention softmax can be modified using proportional attention, i.e., $A = \mathrm{softmax}(QK^{\top}/\sqrt{d} + \log s)$, where $s$ tracks the size or multiplicity of each token [2210.09461].
- **Adaptive and Layerwise Schedules:** Merging can be uniformly distributed across layers, concentrated in early/mid/late blocks, or dynamically adapted per layer or per input sequence via multi-objective optimization or importance signals [2509.09168] [2411.16720].

## 2. Representative Methods and Algorithms

Several algorithmic approaches have been introduced and studied in the literature:

| Method              | Merge Criterion         | Merge Grouping / Schedule      | Notable Feature                |
|---------------------|------------------------|-------------------------------|--------------------------------|
| ToMe [2210.09461]   | Cosine similarity (keys) | Bipartite soft matching, fixed # per layer | Fast, highly parallel, no training |
| ATC [2409.11923]    | Cosine distance         | Agglomerative hierarchical clustering | Superior at low keep rates     |
| PiToMe [2405.16148] | Cluster “energy” (graph) | Preserves low-energy tokens, merges high-energy clusters | Spectrum preservation, informative token retention |
| QuickMerge++ [2508.13204] | Attention entropy       | Entropy-based budgeting, AR prior     | AR compatible, salience weighted     |
| ALGM [2406.09936]   | Cosine similarity       | Two-stage: local early, global mid    | Semantic segmentation, adaptivity    |
| CubistMerge [2509.21764] | Local path graph         | 2D reduction, local bipartite         | Preserves strict spatial grid       |
| DTEM [2412.10569]   | Learned decoupled embedding | Differentiable relaxed matching  | End-to-end trainable grouping       |
| MergeDNA [2511.14806]| Local window similarity    | Hierarchical, context-aware          | Dynamic tokenization for DNA         |

There exist numerous domain-specific variants, e.g., Co-Me for geometric transformers (confidence-guided) [2511.14751], A-ToMe for adjacent token merging in speech [2306.16009], and VQ-integrated methods such as MergeVQ for masked image modeling [2504.00999].

## 3. Integration with Model Architectures and Tasks

Token merging is applicable, with minimal modifications, to a broad range of model families and tasks:

- **Vision Transformers and Dense Tasks:** Merging is used both for classification (e.g., ViT, DeiT), where global similarity suffices, and for dense prediction (segmentation, object detection), where spatial structure and local/semantic detail must be preserved [2406.09936] [2510.01948].
- **Structured and Spatial ViTs:** For models relying on windowed attention or 2D positional priors (e.g., Swin, SAM, DINOv3), spatially-structured merging (e.g., CubistMerge: row/column reduction, local matching) is necessary to ensure compatibility with window partitioning and grid-based biases [2509.21764].
- **Language and Code Models:** In code, merging subtokens forming a semantic unit (BPE fragments of identifiers) by averaging or attention-weighted sum compresses sequence length without retraining the backbone [2507.14423].
- **Time Series, Genomics, and SSMs:** Local window-constrained merging (e.g., in MergeDNA for DNA, or local/causal merging for time series) maintains linear complexity while providing learned, data-dependent compression [2405.17951] [2511.14806] [2508.13599].
- **Autoregressive and Diffusion Models:** Importance-guided merging (via classifier-free guidance in diffusion, or attention entropy in AR transformers) allows dynamic token budgeting and consistent generation quality [2411.16720] [2508.13204].
- **Semantic Communication:** Token merging with layerwise, Pareto-optimized budgets enables runtime adaptation to system constraints, such as wireless SNR, and efficient on-device inference [2509.09168].

## 4. Empirical Benefits and Performance Trade-offs

Token merging consistently delivers substantial reductions in computational cost, memory usage, and latency, with minimal or even positive effects on task performance:

- **Throughput Gains:** ToMe achieves $2\times$–$2.2\times$ throughput on ViT-L and ViT-H, and $1.7\times$ on DeiT-S, with $<0.5\%$ accuracy loss [2210.09461]. ATC offers superior accuracy retention at high merge rates (e.g., $+9.6$pp over ToMe at $r=25\%$ keep on NABirds) [2409.11923].
- **Semantic Segmentation Speedup:** ALGM improves throughput up to $2\times$ on ADE20K, with $+1.1$ mIoU and adaptive trade-off [2406.09936]. Segformer++ achieves $1.61\times$–$1.94\times$ speedup on Cityscapes at $<0.4$ mIoU drop [2405.14467].
- **Fine Detail Preservation:** IBTM outperforms ToMeSD in image/video generation, preserving high-information regions and improving FID and LPIPS metrics, especially under aggressive token reduction (e.g., FID $20.89\rightarrow17.75$ at $r=0.75$) [2411.16720].
- **Domain-Specific Benefits:** In ASR, A-ToMe reduces token count by $57\%$, $1.7\times$ GPU speedup, and $0.01$ WER degradation [2306.16009]. MergeDNA achieves $\sim 3 \times$ quadratic cost reduction and new SOTA on DNA benchmarks [2511.14806]. ClustViT yields $2.2\times$ fewer GFLOPs and $1.6\times$ faster segmentation with $\leq 3.7\%$ mIoU loss [2510.01948].

## 5. Domain and Task-Specific Variants

Recent research has proposed merging schemes tailored to the constraints of specific model classes:

- **SSM-Based Vision Models:** MaMe exploits the SSM state-transition Δ as an informativeness measure, penalizing merges across highly informative tokens to maintain sequential modeling fidelity [2508.13599].
- **Dense Prediction and Segmentation:** Two-stage and semantically-supervised merges (ALGM, ClustViT) combine local or mask-guided clustering with unmerging or regeneration steps, reliably preserving boundary detail and spatial coverage while accelerating computation [2406.09936] [2510.01948].
- **Spatial ViTs:** CubistMerge enforces reduced token grids, local bipartite matching, and component-wise max fusion to maintain compatibility with windowed or RoPE-based models, enabling token count reduction without sacrificing positional bias or spatial structure [2509.21764].
- **Genomics and Long Sequences:** MergeDNA stacks differentiable local merging layers to induce a data-driven, dynamic tokenizer, with joint sequence chunking and pretraining under merged token reconstruction and adaptive masking objectives [2511.14806].

## 6. Limitations, Challenges, and Future Directions

While token merging is widely effective, several open issues and limitations remain:

- **Information Loss at High Merge Rates:** Aggressive merging can induce noticeable performance degradation, especially in tasks requiring fine-grained details (e.g., detailed reconstruction, dense segmentation). Spectral analyses (PiToMe, MergeDNA) and norm-preserving fusion (MLERP) aim to mitigate this [2405.16148] [2511.14806] [2312.01026].
- **Compatibility with Non-Standard Architectures:** Some spatial ViT variants or autoregressive decoders require specially structured merging to maintain attention mask or spatial layout invariants [2509.21764] [2508.13204].
- **Dynamic or Learned Budgets:** Most methods use static schedules or hyperparameters; active research investigates Bayesian optimization of per-layer budgets (edge ViTs, semantic communication) and adaptive policies based on input statistics (entropy, attention, importance) [2509.09168] [2508.13204] [2411.16720].
- **Extensibility to Arbitrary Modalities:** Existing merging relies primarily on similarity in key or embedding space; several works propose integrating geometric, spatial, or downstream-task priors (ToSA with spatial tokens; ClustViT with pseudo-cluster labels) [2506.20066] [2510.01948].

## 7. Impact and Research Trajectory

Token merging has rapidly transitioned from a simple off-the-shelf plug-in (ToMe) to a versatile, domain-adaptive, and theoretically principled ecosystem of algorithms, with strong empirical utility across vision, language, speech, time series, genomics, and generative modeling. Research trends point toward:

- End-to-end differentiable and feature-decoupled merging [2412.10569]
- Energy and importance-aware policies preserving task-critical tokens [2405.16148] [2411.16720]
- Integration with quantization, clustering, and task-specific regeneration [2504.00999] [2510.01948]
- Robustness to extreme compression and application to resource-constrained deployment [2509.09168],[2411.16720]
- Application to adaptive and hierarchical tokenization in non-canonical domains [2511.14806],[2405.17951]

This convergence of efficient inference, spectral/structural preservation, and downstream task robustness positions token merging as a central technology in the next generation of efficient transformer and sequence models.

Source: https://www.emergentmind.com/topics/token-merging