---
title: Adaptive Token Fusion Overview
url: https://www.emergentmind.com/topics/adaptive-token-fusion-atf
type: topic
---

# Adaptive Token Fusion Overview

Adaptive Token Fusion (ATF) denotes a family of token-level mechanisms in which fusion is conditioned on the current input, layer, timestep, or modality rather than fixed a priori. In the recent literature, the term appears explicitly in composed image retrieval, but closely related methods are introduced under other names, including TokenFusion, Token Fusion (ToFu), Contextual Token Fusion, Sparse Temporal Token Fusion, Mixture of States, Adaptive Token Merging, Active Token Mixer, and Adaptive Token Dictionary. Taken together, these works indicate that ATF is better understood as a design space spanning token replacement, merging, routing, caching, grouping, and multimodal aggregation than as a single canonical operator [2504.10995] [2204.08721] [2312.01026] [2506.22396] [2511.18504] [2511.12207] [2509.09955] [2203.06108] [2603.02581].

## 1. Conceptual scope and naming

The term ATF is not standardized across arXiv literature. In some works, adaptivity is explicit and named, as in TMCIR’s “Adaptive Token Fusion,” where image and text tokens are matched by cosine similarity and fused only when their similarity exceeds a threshold [2504.10995]. In other works, the same design impulse appears under different labels. TokenFusion for multimodal vision dynamically detects uninformative tokens and replaces them with aligned features from another modality, which is adaptive at the token level but framed as cross-modal token substitution rather than ATF [2204.08721]. ToFu, by contrast, treats token reduction as a hybrid of pruning and merging, and argues that the appropriate fusion operator depends on layerwise functional linearity [2312.01026].

A second naming axis concerns where adaptation occurs. Some methods are intra-sequence token mergers. QuickSilver’s Contextual Token Fusion merges adjacent or locally connected LLM tokens online during inference, after representations become semantically stable enough for similarity-based contraction [2506.22396]. Other methods are explicitly temporal. Sparse Temporal Token Fusion (STTF) in edge vision-language modeling reuses tokens across frames, updates only changed regions, and fuses temporally redundant representations by cosine similarity thresholding [2511.18504]. Still others move away from direct token merging toward state routing. Mixture of States (MoS) fuses hidden states associated with tokens across layers of an understanding tower, using token-wise top-$k$ routing conditioned on the current latent state and denoising timestep [2511.12207].

A third axis concerns whether “fusion” means efficiency-oriented compression or representational enrichment. In semantic communication, adaptive token merging is used to reduce both edge compute and transmitted semantic payload [2509.09955]. In image restoration, Adaptive Token Dictionary combines dictionary-based cross-attention, token categorization, grouped self-attention, and category-aware FFN fusion to recover global interactions with linear complexity in image size [2603.02581]. In saliency prediction, DTFSal uses learnable token enhancement, shift-based token fusion, and adaptive multimodal fusion, emphasizing representational refinement rather than hard token reduction [2504.10070]. This diversity suggests that ATF is a broad methodological category defined by adaptive token-level combination rules, not by a single architectural lineage.

## 2. Major mechanism families

The literature organizes naturally into a small number of recurrent ATF mechanisms.

| Mechanism family | Core operation | Representative papers |
|---|---|---|
| Cross-modal replacement | Replace low-information tokens with aligned tokens from another modality | [2204.08721] |
| Similarity-based merging | Merge redundant tokens by thresholded similarity and weighted averaging | [2312.01026], [2506.22396], [2509.09955] |
| Temporal reuse and fusion | Reuse cached tokens across timesteps and update only changed regions | [2511.18504] |
| Router-based state fusion | Route each token to a sparse mixture of hidden states across layers | [2511.12207] |
| Prototype or dictionary-guided grouping | Fuse tokens through learned external memory and category assignment | [2603.02581] |
| Channel/spatial adaptive token mixing | Recompose and fuse tokens through offsets, shifts, and channel-wise gates | [2203.06108], [2504.10070] |

The first family is adaptive cross-modal substitution. TokenFusion computes token informativeness scores and applies a thresholded keep-or-replace rule. A token in modality \(m\) is retained if its score exceeds \(\theta\); otherwise it is replaced by a projected aligned token from modality \(m'\). The core update is
\[
\bm{e}_m^l=
\bm{e}_m^l\odot\mathbb{I}_{s^l(\bm{e}_m^{l})\ge\theta}
+\mathrm{Proj}^\text{M}_{m'}(\bm{e}_m^l)\odot\mathbb{I}_{s^l(\bm{e}_m^{l})<\theta},
\]
which makes ATF a hard, alignment-aware substitution rule rather than a soft weighted average [2204.08721].

The second family is similarity-based token merging. ToFu inherits Bipartite Soft Matching from ToMe, but applies pruning-like reduction in early layers and averaging or MLERP merging in later layers, reflecting the claim that shallow layers are less tolerant to interpolated token features than deeper ones [2312.01026]. QuickSilver performs contextual token fusion in frozen LLMs during inference, using local hidden-state \(L_2\) distances, adjacency constraints, and an average or weighted-average super-token that persists through later layers [2506.22396]. The semantic communication framework in adaptive token merging likewise uses per-layer cosine-similarity thresholds over Value vectors to decide whether source tokens should be merged into destination tokens, and aggregates them by norm-weighted averaging [2509.09955].

The third family is temporal ATF. STTF is not only a token fusion module but a token reuse policy over time. It performs event-driven change detection, sparse patch extraction, selective token updates from memory, and thresholded fusion of temporally similar tokens. In ATF terms, it combines adaptive token selection, token reuse/caching, temporal matching, and learned gated averaging [2511.18504].

The fourth family is state-routing fusion. MoS does not merge tokens into fewer tokens; instead it fuses multiple candidate hidden states for each token across source layers. For each token and generation block, a router predicts logits over understanding-tower layers, applies column-wise softmax, selects a top-$k$ subset with \(\epsilon\)-greedy exploration, and aggregates the selected hidden states as a weighted sum. This is best described as adaptive token-conditioned state fusion rather than classical token merging [2511.12207].

The fifth family is memory-guided grouping and fusion. ATD’s Token Dictionary Cross-Attention (TDCA) lets each image token query a learned dictionary, then uses the resulting attention map both to retrieve external priors and to assign tokens to categories. Adaptive Category-based Self-Attention (ACMSA) then performs self-attention within fixed-size sub-categories rather than within local windows, and Category-aware FFN (CFFN) injects the selected dictionary entry back into the feed-forward pathway [2603.02581].

A sixth family emphasizes efficient token mixing rather than explicit sequence shortening. Active Token Mixer predicts channel-wise offsets from each query token, samples horizontal and vertical context per channel, and fuses horizontal, vertical, and identity branches by channel-wise softmax weights [2203.06108]. DTFSal’s LTEB and DLTFB use input-dependent token weighting, learned token banks, and shift-based token reorganization to emphasize salient cues and enlarge effective receptive fields without quadratic attention [2504.10070].

## 3. Canonical operators and mathematical patterns

Despite their diversity, ATF methods reuse a small set of mathematical motifs.

A first motif is thresholded similarity gating. In STTF, temporal redundancy is formalized by
\[
\mathcal{T}_t = \text{Fuse}(\{x_i \in \mathcal{T}_{t-1} \mid \cos(x_i, x_j) > \tau, \, x_j \in \mathcal{T}_t \}),
\]
so fusion occurs when previous and current tokens exceed a cosine-similarity threshold [2511.18504]. In adaptive token merging for semantic communication, the same logic appears as
\[
\text{Merge}(a \to b^*) \text{ if } s_{ab^*}^{(l)} > \tau_l,
\]
with \(s_{ab}^{(l)}\) computed from cosine similarity of Value vectors and \(\tau_l\) chosen per layer [2509.09955]. QuickSilver uses an analogous trigger based on hidden-state \(L_2\) distance and locality constraints, but applies it only after halting decisions have already filtered the active set [2506.22396].

A second motif is weighted aggregation of matched tokens or states. TMCIR computes image-text similarities
\[
\mathbf{S}_{ij} = \frac{\mathbf{v}_i \cdot \mathbf{t}_j}{\|\mathbf{v}_i\| \cdot \|\mathbf{t}_j\|},
\]
then fuses matched pairs by
\[
\mathbf{f}_{i,j} = \frac{\mathbf{S}_{ij} \cdot \mathbf{v}_i + \mathbf{S}_{ij} \cdot \mathbf{t}_j}{2\mathbf{S}_{ij} + \epsilon} + 0.5 \cdot \left(\mathbf{P}_\text{img}[i] + \mathbf{P}_\text{txt}[j]\right),
\]
while retaining unmatched tokens with positional residuals [2504.10995]. MoS uses the same weighted-sum principle at the state level:
\[
\mathbf S_j^c = \sum_{i\in I_j} \overline w_{ij}\cdot \mathcal S_i^c,
\]
where \(I_j\) is a top-$k$ subset of source layers chosen for generation block \(j\) [2511.12207]. In adaptive token merging for semantic communication, the fused representative is a norm-weighted average over a destination token and all source tokens assigned to it [2509.09955].

A third motif is hard routing or hard assignment followed by local interaction. TokenFusion uses a score threshold \(\theta\) to decide whether a token remains self-modal or is overwritten by aligned cross-modal content [2204.08721]. ATD assigns each token to a category by the dictionary entry with maximal TDCA attention,
\[
\bm{\theta}^i = \{ \bm{x}_j \mid \operatorname{arg}\operatorname{max}_k (\bm{A}_D^{jk}) = i\},
\]
then builds fixed-size sub-categories for grouped attention [2603.02581]. This suggests that a large fraction of ATF designs use soft similarity to infer structure, but eventually commit to discrete grouping, replacement, or top-$k$ routing decisions.

A fourth motif is channel-wise or branch-wise adaptive fusion. Active Token Mixer recomposes tokens through horizontal and vertical sampling and then fuses the resulting branches using channel-wise coefficients \(\bm{\alpha}^W,\bm{\alpha}^H,\bm{\alpha}^I\) [2203.06108]. DTFSal’s Learnable Token Enhancement Block similarly produces soft weights over a learnable token bank, builds an aggregated token map, and reinjects it by residual modulation, while its Adaptive Multimodal Fusion Block computes stream weights over local, global, and deformable branches [2504.10070]. These mechanisms show that ATF need not reduce sequence length; it may instead adaptively reweight the content that each token receives.

## 4. Architectural placement and optimization regimes

ATF mechanisms occupy markedly different positions in model pipelines. Some are front-end modules that alter token formation before the main backbone. Early token fusion for image classification constructs an image-like tensor by combining multiple ResNet stages through UpConv and \(1\times1\) convolution, then patchifies the fused result for transformer processing [2208.09183]. TokenFusion for multimodal vision is inserted before each transformer layer, so adaptive replacement happens repeatedly while the backbone architecture remains largely intact [2204.08721].

Other methods intervene inside transformer blocks. ToFu places a token reduction module \(R\) inside transformer blocks, illustrated before the MLP, so that token count decreases progressively through depth [2312.01026]. Adaptive token merging for semantic communication merges the corresponding hidden states after attention and before the feed-forward neural network [2509.09955]. QuickSilver activates contextual token fusion in deeper layers, from layer 12 onward in the implementation description, after contextualization has made hidden-state similarity more meaningful [2506.22396].

Temporal and multimodal generation models extend placement beyond a single depth axis. STTF sits between event-driven change detection and multimodal decoding: an EventGateCNN produces a change mask, active patches are extracted, token memory performs selective update, and the updated token set is then consumed by temporal cross-attention with text [2511.18504]. MoS inserts a router between an understanding tower and a generation tower; for each denoising step and generation block, the router predicts which understanding-layer states should be aggregated and injected [2511.12207]. ATD integrates TDCA, ACMSA, shifted-window attention, and CFFN inside each restoration layer, reusing a shared dictionary across multiple layers in a block [2603.02581].

Optimization regimes are equally heterogeneous. Several methods are explicitly training-free. ToFu is designed as a plug-in reduction method usable with or without additional training, and its presented experiments emphasize off-the-shelf inference acceleration [2312.01026]. QuickSilver requires no retraining, no architectural change, and no auxiliary networks [2506.22396]. Adaptive token merging for semantic communication is training-free and instead optimizes layerwise thresholds offline by multi-objective Bayesian optimization over accuracy, FLOPs, and communication cost [2509.09955]. SEATS for omni-modal LLMs is likewise training-free and uses stage-adaptive selection driven by attention-weighted diversity and query relevance rather than learned token fusers [2605.20035].

By contrast, other methods are jointly trained with task losses and sparsity or routing objectives. TokenFusion uses task-specific losses plus an \(l_1\) penalty on token informativeness scores [2204.08721]. STTF and ANC are jointly trained with a composite loss containing a task term, a token-count sparsity term, and an ANC activation sparsity term, although the printed formula is typographically malformed [2511.18504]. TMCIR applies ATF only after a first-stage alignment procedure and then fine-tunes all encoders contrastively against target images [2504.10995]. MoS trains the router end-to-end under the rectified-flow objective, without an auxiliary routing loss, relying instead on top-$k$ sparse selection with \(\epsilon\)-greedy exploration [2511.12207].

## 5. Empirical behavior across tasks and domains

ATF-style mechanisms have been evaluated in a wide range of settings, from edge vision-language models to classification, retrieval, restoration, generation, and saliency prediction. In edge vision-language modeling, STTF reports one of the clearest efficiency-accuracy trade-offs: average token count drops from 196 to approximately 31, corresponding to an \(84\%\) reduction, while accuracy on DVS128 Gesture remains \(95.6\%\). The same work reports a \(6.1\times\) speedup over dense ViT-based baselines, TinyGPT-STTF captioning quality of CIDEr \(131.2\), BLEU-4 \(0.38\), METEOR \(0.31\), and ROUGE-L \(0.56\), and an abstract-level claim of \(62\times\) fewer on-device FLOPs than LLaVA-1.5 7B [2511.18504].

In frozen LLM inference, QuickSilver attributes a substantial portion of its runtime gains to contextual token fusion. The full stack achieves up to \(39.6\%\) FLOP reduction with negligible perplexity degradation \((\le 0.2)\), while an isolated appendix ablation assigns token fusion roughly \(23\%\) speedup, \(30\%\) FLOPs reduction, and \(+0.18\) perplexity relative to dense inference [2506.22396]. Its syntactic safety analysis further reports that fused pairs lie in the same syntactic chunk with \(78.9\%\), \(81.2\%\), and \(84.5\%\) precision at layers 12, 15, and 20, respectively, against a random adjacency baseline around \(47\%\)–\(50\%\) [2506.22396].

In multimodal retrieval, TMCIR is one of the few systems to name ATF explicitly and to isolate its contribution. Removing token merging drops Fashion-IQ from R@10 \(56.57\) and R@50 \(76.55\) to \(29.68\) and \(54.85\); on CIRR, R@1 falls from \(54.12\) to \(20.88\), R@5 from \(84.27\) to \(48.24\), and \(R_{\text{subset}@1}\) from \(82.64\) to \(50.33\) [2504.10995]. The same paper reports that a similarity threshold of \(0.7\) gives the best trade-off, with lower thresholds admitting noisy matches and higher thresholds discarding useful ones [2504.10995].

In multimodal vision transformers, TokenFusion reports strong results across image-to-image translation, RGB-depth segmentation, and heterogeneous 3D detection. On NYUDv2 segmentation, concatenation improves from \(51.4\) mIoU to \(54.2\) mIoU with TokenFusion; on SUN RGB-D, from \(49.0\) to \(53.0\) mIoU; and on ScanNetV2 3D detection, TokenFusion improves over Group-Free and naive RGB appending, reaching \(70.8/54.2\) mAP@0.25/@0.5 [2204.08721]. Its ablations show that learned token substitution plus Residual Positional Alignment outperforms random fusion and \(l_1\)-only variants, indicating that adaptive token-level decisions rather than arbitrary mixing are responsible for the gains [2204.08721].

For training-free ViT compression, ToFu consistently improves on ToMe under matched FLOPs. On ViT-B/ImageNet at \(r=16\), ToMe reports Top-1 \(78.88\), whereas ToFu AVG reaches \(80.43\) and ToFu MLERP \(80.70\); at \(r=20\), the gap widens from \(67.54\) for ToMe to \(74.06\) for ToFu MLERP [2312.01026]. In Stable Diffusion v1.5, ToFu improves FID from \(15.74\) to \(14.72\), LPIPS from \(0.3133\) to \(0.2706\), and MS-SSIM from \(0.7304\) to \(0.7618\) at nearly identical time and memory [2312.01026].

Router-based ATF alternatives also show strong empirical effects. MoS reports that token-wise routing beats sample-wise routing, with FID \(20.17\) and CLIP \(21.63\) versus FID \(21.66\) and CLIP \(21.48\), and that prompt + latent + timestep routing outperforms prompt-only or prompt + latent configurations on MJHQ [2511.12207]. In text-to-image evaluation, MoS-S and MoS-L report GenEval \(0.89/0.90\) and DPG \(86.33/87.01\), while the paper also states that adaptive routing outperforms a hand-crafted routing baseline with FID \(17.77\) versus \(21.51\) and CLIP \(22.91\) versus \(22.04\) [2511.12207].

Prototype- and group-based ATF in image restoration shows a different empirical pattern. In ATD-light \(\times4\) ablations, adding TDCA improves Urban100/Manga109 from \(26.40/30.84\) to \(26.46/30.92\), adding ACMSA further improves to \(26.70/31.19\), and adding CFFN reaches \(26.76/31.21\), indicating that adaptive grouping and category-aware fusion contribute incrementally beyond dictionary retrieval alone [2603.02581]. The full ATD achieves \(28.24\) on Urban100 and \(32.71\) on Manga109 at \(\times4\), exceeding HAT and MambaIRv2 on those benchmarks [2603.02581].

In saliency prediction, DTFSal reports that AMFB outperforms both concatenation and cross-attention baselines. On ETMD, AMFB reaches SIM \(0.531\), CC \(0.667\), NSS \(3.82\), and AUC-J \(0.957\), compared with \(0.381/0.625/2.74/0.910\) for concatenation and \(0.485/0.635/3.16/0.907\) for cross-attention [2504.10070]. Its stage ablation further indicates that DLTFB is most effective at stage 4, where CC reaches \(0.561\), NSS \(3.205\), AUC-J \(0.923\), and SIM \(0.446\) [2504.10070].

## 6. Limitations, ambiguities, and open directions

A central limitation of ATF as a research area is definitional. Several papers are highly relevant to ATF while explicitly noting that they do not use the term. QuickSilver presents “Contextual Token Fusion,” not “Adaptive Token Fusion” [2506.22396]. STTF is described as an ATF-style mechanism but not explicitly labeled that way [2511.18504]. MoS is a close conceptual match only if ATF is interpreted broadly enough to include adaptive token-conditioned state routing rather than literal token merging [2511.12207]. SEATS, finally, is best understood as fusion-aware adaptive pruning rather than token fusion proper, because its central operation is retention and removal rather than aggregation [2605.20035]. This suggests that the boundary of ATF is still contested.

A second limitation is under-specification. STTF states that fusion is performed via learned gated averaging and that the threshold \(\tau\) is adapted per layer using a lightweight policy network trained with latency regularization, but it provides no formula or architecture for either component [2511.18504]. TMCIR leaves open whether image-text matching is one-to-one, one-to-many, or many-to-many, and its printed loss equations contain indexing inconsistencies [2504.10995]. Famba-V describes cosine-similarity-based matching and cross-layer scheduling heuristics but does not specify the exact matching algorithm beyond selecting the \(r\) most similar pairs [2409.09808]. Such gaps make cross-paper ATF comparisons more difficult than headline results may suggest.

A third limitation is generalization and semantic safety. STTF reports severe overfitting: training accuracy approaches \(98\%\), while validation accuracy peaks near \(38\%\) and then plateaus, leading the authors to state that aggressive sparsity induction disrupts generalization unless paired with strong regularization [2511.18504]. QuickSilver discusses “context bleed,” where locally similar tokens should remain distinct because of sentence-level role differences, and proposes contextual divergence filters and exclusion policies for named entities, punctuation, or sensitive terms [2506.22396]. TokenFusion depends heavily on reliable inter-modal alignment, whether same-position correspondence for homogeneous modalities or camera geometry for heterogeneous ones, so noisy alignment can turn adaptive substitution into systematic feature corruption [2204.08721].

A fourth limitation is that not all adaptive policies are fully learned or fully sample-conditional. ToFu’s layer schedule is fixed by depth even though token matching itself is content-dependent [2312.01026]. Famba-V uses content-based pairing but static layer schedules and fixed merge budgets [2409.09808]. Adaptive token merging for semantic communication searches threshold policies offline rather than learning them jointly with the backbone [2509.09955]. SEATS uses backbone-specific heuristic layer boundaries for late removal of non-text tokens [2605.20035]. A plausible implication is that future ATF systems may combine the robustness of training-free mechanisms with more explicit learned controllers or hardware-aware online policies.

The research trajectory already points toward such hybridization. STTF’s own future-work discussion proposes “Hybrid STTF+ANC Fusion,” combining token caching with adaptive branch routing [2511.18504]. MoS identifies bidirectional routing as future work, which would extend adaptive fusion from understanding-to-generation injection to symmetric co-fusion [2511.12207]. The semantic communication framework suggests adding hardware-aware objectives such as latency or energy, and possibly privacy as a fourth objective in multi-objective optimization [2509.09955]. ATD shows that external prototype memory can drive both fusion and routing, indicating a path toward memory-augmented ATF with explicit category structure [2603.02581]. Together, these directions suggest that ATF is evolving from isolated token-merging heuristics toward broader adaptive-computation systems in which token fusion, routing, grouping, and retention are optimized jointly.

Source: https://www.emergentmind.com/topics/adaptive-token-fusion-atf