Adaptive Token Fusion Overview

• Adaptive Token Fusion (ATF) is a family of adaptive mechanisms that dynamically merge, replace, and route tokens based on content similarity and temporal or cross-modal cues.
• ATF methods span diverse designs, including cross-modal replacement, similarity-based merging, temporal reuse, state-routing fusion, and adaptive token mixing.
• These mechanisms enhance efficiency and accuracy in applications from image retrieval and transformer acceleration to semantic communication and saliency prediction.

Adaptive Token Fusion (ATF) denotes a family of token-level mechanisms in which fusion is conditioned on the current input, layer, timestep, or modality rather than fixed a priori. In the recent literature, the term appears explicitly in composed image retrieval, but closely related methods are introduced under other names, including TokenFusion, Token Fusion (ToFu), Contextual Token Fusion, Sparse Temporal Token Fusion, Mixture of States, Adaptive Token Merging, Active Token Mixer, and Adaptive Token Dictionary. Taken together, these works indicate that ATF is better understood as a design space spanning token replacement, merging, routing, caching, grouping, and multimodal aggregation than as a single canonical operator (Wang et al., 15 Apr 2025, Wang et al., 2022, Kim et al., 2023, Khanna et al., 27 Jun 2025, Tanvir et al., 23 Nov 2025, Liu et al., 15 Nov 2025, Erak et al., 12 Sep 2025, Wei et al., 2022, Zhang et al., 3 Mar 2026).

1. Conceptual scope and naming

The term ATF is not standardized across arXiv literature. In some works, adaptivity is explicit and named, as in TMCIR’s “Adaptive Token Fusion,” where image and text tokens are matched by cosine similarity and fused only when their similarity exceeds a threshold (Wang et al., 15 Apr 2025). In other works, the same design impulse appears under different labels. TokenFusion for multimodal vision dynamically detects uninformative tokens and replaces them with aligned features from another modality, which is adaptive at the token level but framed as cross-modal token substitution rather than ATF (Wang et al., 2022). ToFu, by contrast, treats token reduction as a hybrid of pruning and merging, and argues that the appropriate fusion operator depends on layerwise functional linearity (Kim et al., 2023).

A second naming axis concerns where adaptation occurs. Some methods are intra-sequence token mergers. QuickSilver’s Contextual Token Fusion merges adjacent or locally connected LLM tokens online during inference, after representations become semantically stable enough for similarity-based contraction (Khanna et al., 27 Jun 2025). Other methods are explicitly temporal. Sparse Temporal Token Fusion (STTF) in edge vision-language modeling reuses tokens across frames, updates only changed regions, and fuses temporally redundant representations by cosine similarity thresholding (Tanvir et al., 23 Nov 2025). Still others move away from direct token merging toward state routing. Mixture of States (MoS) fuses hidden states associated with tokens across layers of an understanding tower, using token-wise top-$k$ routing conditioned on the current latent state and denoising timestep (Liu et al., 15 Nov 2025).

A third axis concerns whether “fusion” means efficiency-oriented compression or representational enrichment. In semantic communication, adaptive token merging is used to reduce both edge compute and transmitted semantic payload (Erak et al., 12 Sep 2025). In image restoration, Adaptive Token Dictionary combines dictionary-based cross-attention, token categorization, grouped self-attention, and category-aware FFN fusion to recover global interactions with linear complexity in image size (Zhang et al., 3 Mar 2026). In saliency prediction, DTFSal uses learnable token enhancement, shift-based token fusion, and adaptive multimodal fusion, emphasizing representational refinement rather than hard token reduction (Hooshanfar et al., 14 Apr 2025). This diversity suggests that ATF is a broad methodological category defined by adaptive token-level combination rules, not by a single architectural lineage.

2. Major mechanism families

The literature organizes naturally into a small number of recurrent ATF mechanisms.

Mechanism family	Core operation	Representative papers
Cross-modal replacement	Replace low-information tokens with aligned tokens from another modality	(Wang et al., 2022)
Similarity-based merging	Merge redundant tokens by thresholded similarity and weighted averaging	(Kim et al., 2023, Khanna et al., 27 Jun 2025, Erak et al., 12 Sep 2025)
Temporal reuse and fusion	Reuse cached tokens across timesteps and update only changed regions	(Tanvir et al., 23 Nov 2025)
Router-based state fusion	Route each token to a sparse mixture of hidden states across layers	(Liu et al., 15 Nov 2025)
Prototype or dictionary-guided grouping	Fuse tokens through learned external memory and category assignment	(Zhang et al., 3 Mar 2026)
Channel/spatial adaptive token mixing	Recompose and fuse tokens through offsets, shifts, and channel-wise gates	(Wei et al., 2022, Hooshanfar et al., 14 Apr 2025)

The first family is adaptive cross-modal substitution. TokenFusion computes token informativeness scores and applies a thresholded keep-or-replace rule. A token in modality $m$ is retained if its score exceeds $\theta$; otherwise it is replaced by a projected aligned token from modality $m'$. The core update is

$$\bm{e}_m^l= \bm{e}_m^l\odot\mathbb{I}_{s^l(\bm{e}_m^{l})\ge\theta} +\mathrm{Proj}^\text{M}_{m'}(\bm{e}_m^l)\odot\mathbb{I}_{s^l(\bm{e}_m^{l})<\theta},$$

which makes ATF a hard, alignment-aware substitution rule rather than a soft weighted average (Wang et al., 2022).

The second family is similarity-based token merging. ToFu inherits Bipartite Soft Matching from ToMe, but applies pruning-like reduction in early layers and averaging or MLERP merging in later layers, reflecting the claim that shallow layers are less tolerant to interpolated token features than deeper ones (Kim et al., 2023). QuickSilver performs contextual token fusion in frozen LLMs during inference, using local hidden-state $L_2$ distances, adjacency constraints, and an average or weighted-average super-token that persists through later layers (Khanna et al., 27 Jun 2025). The semantic communication framework in adaptive token merging likewise uses per-layer cosine-similarity thresholds over Value vectors to decide whether source tokens should be merged into destination tokens, and aggregates them by norm-weighted averaging (Erak et al., 12 Sep 2025).

The third family is temporal ATF. STTF is not only a token fusion module but a token reuse policy over time. It performs event-driven change detection, sparse patch extraction, selective token updates from memory, and thresholded fusion of temporally similar tokens. In ATF terms, it combines adaptive token selection, token reuse/caching, temporal matching, and learned gated averaging (Tanvir et al., 23 Nov 2025).

The fourth family is state-routing fusion. MoS does not merge tokens into fewer tokens; instead it fuses multiple candidate hidden states for each token across source layers. For each token and generation block, a router predicts logits over understanding-tower layers, applies column-wise softmax, selects a top-$k$ subset with $\epsilon$-greedy exploration, and aggregates the selected hidden states as a weighted sum. This is best described as adaptive token-conditioned state fusion rather than classical token merging (Liu et al., 15 Nov 2025).

The fifth family is memory-guided grouping and fusion. ATD’s Token Dictionary Cross-Attention (TDCA) lets each image token query a learned dictionary, then uses the resulting attention map both to retrieve external priors and to assign tokens to categories. Adaptive Category-based Self-Attention (ACMSA) then performs self-attention within fixed-size sub-categories rather than within local windows, and Category-aware FFN (CFFN) injects the selected dictionary entry back into the feed-forward pathway (Zhang et al., 3 Mar 2026).

A sixth family emphasizes efficient token mixing rather than explicit sequence shortening. Active Token Mixer predicts channel-wise offsets from each query token, samples horizontal and vertical context per channel, and fuses horizontal, vertical, and identity branches by channel-wise softmax weights (Wei et al., 2022). DTFSal’s LTEB and DLTFB use input-dependent token weighting, learned token banks, and shift-based token reorganization to emphasize salient cues and enlarge effective receptive fields without quadratic attention (Hooshanfar et al., 14 Apr 2025).

3. Canonical operators and mathematical patterns

Despite their diversity, ATF methods reuse a small set of mathematical motifs.

A first motif is thresholded similarity gating. In STTF, temporal redundancy is formalized by

$$\mathcal{T}_t = \text{Fuse}(\{x_i \in \mathcal{T}_{t-1} \mid \cos(x_i, x_j) > \tau, \, x_j \in \mathcal{T}_t \}),$$

so fusion occurs when previous and current tokens exceed a cosine-similarity threshold (Tanvir et al., 23 Nov 2025). In adaptive token merging for semantic communication, the same logic appears as

$$\text{Merge}(a \to b^*) \text{ if } s_{ab^*}^{(l)} > \tau_l,$$

with $m$0 computed from cosine similarity of Value vectors and $m$1 chosen per layer (Erak et al., 12 Sep 2025). QuickSilver uses an analogous trigger based on hidden-state $m$2 distance and locality constraints, but applies it only after halting decisions have already filtered the active set (Khanna et al., 27 Jun 2025).

A second motif is weighted aggregation of matched tokens or states. TMCIR computes image-text similarities

$m$3

then fuses matched pairs by

$m$4

while retaining unmatched tokens with positional residuals (Wang et al., 15 Apr 2025). MoS uses the same weighted-sum principle at the state level: $m$5 where $m$6 is a top-$m$7 subset of source layers chosen for generation block $m$8 (Liu et al., 15 Nov 2025). In adaptive token merging for semantic communication, the fused representative is a norm-weighted average over a destination token and all source tokens assigned to it (Erak et al., 12 Sep 2025).

A third motif is hard routing or hard assignment followed by local interaction. TokenFusion uses a score threshold $m$9 to decide whether a token remains self-modal or is overwritten by aligned cross-modal content (Wang et al., 2022). ATD assigns each token to a category by the dictionary entry with maximal TDCA attention,

$\theta$0

then builds fixed-size sub-categories for grouped attention (Zhang et al., 3 Mar 2026). This suggests that a large fraction of ATF designs use soft similarity to infer structure, but eventually commit to discrete grouping, replacement, or top-$\theta$1 routing decisions.

A fourth motif is channel-wise or branch-wise adaptive fusion. Active Token Mixer recomposes tokens through horizontal and vertical sampling and then fuses the resulting branches using channel-wise coefficients $\theta$2 (Wei et al., 2022). DTFSal’s Learnable Token Enhancement Block similarly produces soft weights over a learnable token bank, builds an aggregated token map, and reinjects it by residual modulation, while its Adaptive Multimodal Fusion Block computes stream weights over local, global, and deformable branches (Hooshanfar et al., 14 Apr 2025). These mechanisms show that ATF need not reduce sequence length; it may instead adaptively reweight the content that each token receives.

4. Architectural placement and optimization regimes

ATF mechanisms occupy markedly different positions in model pipelines. Some are front-end modules that alter token formation before the main backbone. Early token fusion for image classification constructs an image-like tensor by combining multiple ResNet stages through UpConv and $\theta$3 convolution, then patchifies the fused result for transformer processing (Choi et al., 2022). TokenFusion for multimodal vision is inserted before each transformer layer, so adaptive replacement happens repeatedly while the backbone architecture remains largely intact (Wang et al., 2022).

Other methods intervene inside transformer blocks. ToFu places a token reduction module $\theta$4 inside transformer blocks, illustrated before the MLP, so that token count decreases progressively through depth (Kim et al., 2023). Adaptive token merging for semantic communication merges the corresponding hidden states after attention and before the feed-forward neural network (Erak et al., 12 Sep 2025). QuickSilver activates contextual token fusion in deeper layers, from layer 12 onward in the implementation description, after contextualization has made hidden-state similarity more meaningful (Khanna et al., 27 Jun 2025).

Temporal and multimodal generation models extend placement beyond a single depth axis. STTF sits between event-driven change detection and multimodal decoding: an EventGateCNN produces a change mask, active patches are extracted, token memory performs selective update, and the updated token set is then consumed by temporal cross-attention with text (Tanvir et al., 23 Nov 2025). MoS inserts a router between an understanding tower and a generation tower; for each denoising step and generation block, the router predicts which understanding-layer states should be aggregated and injected (Liu et al., 15 Nov 2025). ATD integrates TDCA, ACMSA, shifted-window attention, and CFFN inside each restoration layer, reusing a shared dictionary across multiple layers in a block (Zhang et al., 3 Mar 2026).

Optimization regimes are equally heterogeneous. Several methods are explicitly training-free. ToFu is designed as a plug-in reduction method usable with or without additional training, and its presented experiments emphasize off-the-shelf inference acceleration (Kim et al., 2023). QuickSilver requires no retraining, no architectural change, and no auxiliary networks (Khanna et al., 27 Jun 2025). Adaptive token merging for semantic communication is training-free and instead optimizes layerwise thresholds offline by multi-objective Bayesian optimization over accuracy, FLOPs, and communication cost (Erak et al., 12 Sep 2025). SEATS for omni-modal LLMs is likewise training-free and uses stage-adaptive selection driven by attention-weighted diversity and query relevance rather than learned token fusers (Xin et al., 19 May 2026).

By contrast, other methods are jointly trained with task losses and sparsity or routing objectives. TokenFusion uses task-specific losses plus an $\theta$5 penalty on token informativeness scores (Wang et al., 2022). STTF and ANC are jointly trained with a composite loss containing a task term, a token-count sparsity term, and an ANC activation sparsity term, although the printed formula is typographically malformed (Tanvir et al., 23 Nov 2025). TMCIR applies ATF only after a first-stage alignment procedure and then fine-tunes all encoders contrastively against target images (Wang et al., 15 Apr 2025). MoS trains the router end-to-end under the rectified-flow objective, without an auxiliary routing loss, relying instead on top-$\theta$6 sparse selection with $\theta$7-greedy exploration (Liu et al., 15 Nov 2025).

5. Empirical behavior across tasks and domains

ATF-style mechanisms have been evaluated in a wide range of settings, from edge vision-LLMs to classification, retrieval, restoration, generation, and saliency prediction. In edge vision-language modeling, STTF reports one of the clearest efficiency-accuracy trade-offs: average token count drops from 196 to approximately 31, corresponding to an $\theta$8 reduction, while accuracy on DVS128 Gesture remains $\theta$9. The same work reports a $m'$0 speedup over dense ViT-based baselines, TinyGPT-STTF captioning quality of CIDEr $m'$1, BLEU-4 $m'$2, METEOR $m'$3, and ROUGE-L $m'$4, and an abstract-level claim of $m'$5 fewer on-device FLOPs than LLaVA-1.5 7B (Tanvir et al., 23 Nov 2025).

In frozen LLM inference, QuickSilver attributes a substantial portion of its runtime gains to contextual token fusion. The full stack achieves up to $m'$6 FLOP reduction with negligible perplexity degradation $m'$7, while an isolated appendix ablation assigns token fusion roughly $m'$8 speedup, $m'$9 FLOPs reduction, and $$\bm{e}_m^l= \bm{e}_m^l\odot\mathbb{I}_{s^l(\bm{e}_m^{l})\ge\theta} +\mathrm{Proj}^\text{M}_{m'}(\bm{e}_m^l)\odot\mathbb{I}_{s^l(\bm{e}_m^{l})<\theta},$$0 perplexity relative to dense inference (Khanna et al., 27 Jun 2025). Its syntactic safety analysis further reports that fused pairs lie in the same syntactic chunk with $$\bm{e}_m^l= \bm{e}_m^l\odot\mathbb{I}_{s^l(\bm{e}_m^{l})\ge\theta} +\mathrm{Proj}^\text{M}_{m'}(\bm{e}_m^l)\odot\mathbb{I}_{s^l(\bm{e}_m^{l})<\theta},$$1, $$\bm{e}_m^l= \bm{e}_m^l\odot\mathbb{I}_{s^l(\bm{e}_m^{l})\ge\theta} +\mathrm{Proj}^\text{M}_{m'}(\bm{e}_m^l)\odot\mathbb{I}_{s^l(\bm{e}_m^{l})<\theta},$$2, and $$\bm{e}_m^l= \bm{e}_m^l\odot\mathbb{I}_{s^l(\bm{e}_m^{l})\ge\theta} +\mathrm{Proj}^\text{M}_{m'}(\bm{e}_m^l)\odot\mathbb{I}_{s^l(\bm{e}_m^{l})<\theta},$$3 precision at layers 12, 15, and 20, respectively, against a random adjacency baseline around $$\bm{e}_m^l= \bm{e}_m^l\odot\mathbb{I}_{s^l(\bm{e}_m^{l})\ge\theta} +\mathrm{Proj}^\text{M}_{m'}(\bm{e}_m^l)\odot\mathbb{I}_{s^l(\bm{e}_m^{l})<\theta},$$4–$$\bm{e}_m^l= \bm{e}_m^l\odot\mathbb{I}_{s^l(\bm{e}_m^{l})\ge\theta} +\mathrm{Proj}^\text{M}_{m'}(\bm{e}_m^l)\odot\mathbb{I}_{s^l(\bm{e}_m^{l})<\theta},$$5 (Khanna et al., 27 Jun 2025).

In multimodal retrieval, TMCIR is one of the few systems to name ATF explicitly and to isolate its contribution. Removing token merging drops Fashion-IQ from R@10 $$\bm{e}_m^l= \bm{e}_m^l\odot\mathbb{I}_{s^l(\bm{e}_m^{l})\ge\theta} +\mathrm{Proj}^\text{M}_{m'}(\bm{e}_m^l)\odot\mathbb{I}_{s^l(\bm{e}_m^{l})<\theta},$$6 and R@50 $$\bm{e}_m^l= \bm{e}_m^l\odot\mathbb{I}_{s^l(\bm{e}_m^{l})\ge\theta} +\mathrm{Proj}^\text{M}_{m'}(\bm{e}_m^l)\odot\mathbb{I}_{s^l(\bm{e}_m^{l})<\theta},$$7 to $$\bm{e}_m^l= \bm{e}_m^l\odot\mathbb{I}_{s^l(\bm{e}_m^{l})\ge\theta} +\mathrm{Proj}^\text{M}_{m'}(\bm{e}_m^l)\odot\mathbb{I}_{s^l(\bm{e}_m^{l})<\theta},$$8 and $$\bm{e}_m^l= \bm{e}_m^l\odot\mathbb{I}_{s^l(\bm{e}_m^{l})\ge\theta} +\mathrm{Proj}^\text{M}_{m'}(\bm{e}_m^l)\odot\mathbb{I}_{s^l(\bm{e}_m^{l})<\theta},$$9; on CIRR, R@1 falls from $L_2$0 to $L_2$1, R@5 from $L_2$2 to $L_2$3, and $L_2$4 from $L_2$5 to $L_2$6 (Wang et al., 15 Apr 2025). The same paper reports that a similarity threshold of $L_2$7 gives the best trade-off, with lower thresholds admitting noisy matches and higher thresholds discarding useful ones (Wang et al., 15 Apr 2025).

In multimodal vision transformers, TokenFusion reports strong results across image-to-image translation, RGB-depth segmentation, and heterogeneous 3D detection. On NYUDv2 segmentation, concatenation improves from $L_2$8 mIoU to $L_2$9 mIoU with TokenFusion; on SUN RGB-D, from $k$0 to $k$1 mIoU; and on ScanNetV2 3D detection, TokenFusion improves over Group-Free and naive RGB appending, reaching $k$2 mAP@0.25/@0.5 (Wang et al., 2022). Its ablations show that learned token substitution plus Residual Positional Alignment outperforms random fusion and $k$3-only variants, indicating that adaptive token-level decisions rather than arbitrary mixing are responsible for the gains (Wang et al., 2022).

For training-free ViT compression, ToFu consistently improves on ToMe under matched FLOPs. On ViT-B/ImageNet at $k$4, ToMe reports Top-1 $k$5, whereas ToFu AVG reaches $k$6 and ToFu MLERP $k$7; at $k$8, the gap widens from $k$9 for ToMe to $\epsilon$0 for ToFu MLERP (Kim et al., 2023). In Stable Diffusion v1.5, ToFu improves FID from $\epsilon$1 to $\epsilon$2, LPIPS from $\epsilon$3 to $\epsilon$4, and MS-SSIM from $\epsilon$5 to $\epsilon$6 at nearly identical time and memory (Kim et al., 2023).

Router-based ATF alternatives also show strong empirical effects. MoS reports that token-wise routing beats sample-wise routing, with FID $\epsilon$7 and CLIP $\epsilon$8 versus FID $\epsilon$9 and CLIP $$\mathcal{T}_t = \text{Fuse}(\{x_i \in \mathcal{T}_{t-1} \mid \cos(x_i, x_j) > \tau, \, x_j \in \mathcal{T}_t \}),$$0, and that prompt + latent + timestep routing outperforms prompt-only or prompt + latent configurations on MJHQ (Liu et al., 15 Nov 2025). In text-to-image evaluation, MoS-S and MoS-L report GenEval $$\mathcal{T}_t = \text{Fuse}(\{x_i \in \mathcal{T}_{t-1} \mid \cos(x_i, x_j) > \tau, \, x_j \in \mathcal{T}_t \}),$$1 and DPG $$\mathcal{T}_t = \text{Fuse}(\{x_i \in \mathcal{T}_{t-1} \mid \cos(x_i, x_j) > \tau, \, x_j \in \mathcal{T}_t \}),$$2, while the paper also states that adaptive routing outperforms a hand-crafted routing baseline with FID $$\mathcal{T}_t = \text{Fuse}(\{x_i \in \mathcal{T}_{t-1} \mid \cos(x_i, x_j) > \tau, \, x_j \in \mathcal{T}_t \}),$$3 versus $$\mathcal{T}_t = \text{Fuse}(\{x_i \in \mathcal{T}_{t-1} \mid \cos(x_i, x_j) > \tau, \, x_j \in \mathcal{T}_t \}),$$4 and CLIP $$\mathcal{T}_t = \text{Fuse}(\{x_i \in \mathcal{T}_{t-1} \mid \cos(x_i, x_j) > \tau, \, x_j \in \mathcal{T}_t \}),$$5 versus $$\mathcal{T}_t = \text{Fuse}(\{x_i \in \mathcal{T}_{t-1} \mid \cos(x_i, x_j) > \tau, \, x_j \in \mathcal{T}_t \}),$$6 (Liu et al., 15 Nov 2025).

Prototype- and group-based ATF in image restoration shows a different empirical pattern. In ATD-light $$\mathcal{T}_t = \text{Fuse}(\{x_i \in \mathcal{T}_{t-1} \mid \cos(x_i, x_j) > \tau, \, x_j \in \mathcal{T}_t \}),$$7 ablations, adding TDCA improves Urban100/Manga109 from $$\mathcal{T}_t = \text{Fuse}(\{x_i \in \mathcal{T}_{t-1} \mid \cos(x_i, x_j) > \tau, \, x_j \in \mathcal{T}_t \}),$$8 to $$\mathcal{T}_t = \text{Fuse}(\{x_i \in \mathcal{T}_{t-1} \mid \cos(x_i, x_j) > \tau, \, x_j \in \mathcal{T}_t \}),$$9, adding ACMSA further improves to $$\text{Merge}(a \to b^*) \text{ if } s_{ab^*}^{(l)} > \tau_l,$$0, and adding CFFN reaches $$\text{Merge}(a \to b^*) \text{ if } s_{ab^*}^{(l)} > \tau_l,$$1, indicating that adaptive grouping and category-aware fusion contribute incrementally beyond dictionary retrieval alone (Zhang et al., 3 Mar 2026). The full ATD achieves $$\text{Merge}(a \to b^*) \text{ if } s_{ab^*}^{(l)} > \tau_l,$$2 on Urban100 and $$\text{Merge}(a \to b^*) \text{ if } s_{ab^*}^{(l)} > \tau_l,$$3 on Manga109 at $$\text{Merge}(a \to b^*) \text{ if } s_{ab^*}^{(l)} > \tau_l,$$4, exceeding HAT and MambaIRv2 on those benchmarks (Zhang et al., 3 Mar 2026).

In saliency prediction, DTFSal reports that AMFB outperforms both concatenation and cross-attention baselines. On ETMD, AMFB reaches SIM $$\text{Merge}(a \to b^*) \text{ if } s_{ab^*}^{(l)} > \tau_l,$$5, CC $$\text{Merge}(a \to b^*) \text{ if } s_{ab^*}^{(l)} > \tau_l,$$6, NSS $$\text{Merge}(a \to b^*) \text{ if } s_{ab^*}^{(l)} > \tau_l,$$7, and AUC-J $$\text{Merge}(a \to b^*) \text{ if } s_{ab^*}^{(l)} > \tau_l,$$8, compared with $$\text{Merge}(a \to b^*) \text{ if } s_{ab^*}^{(l)} > \tau_l,$$9 for concatenation and $m$00 for cross-attention (Hooshanfar et al., 14 Apr 2025). Its stage ablation further indicates that DLTFB is most effective at stage 4, where CC reaches $m$01, NSS $m$02, AUC-J $m$03, and SIM $m$04 (Hooshanfar et al., 14 Apr 2025).

6. Limitations, ambiguities, and open directions

A central limitation of ATF as a research area is definitional. Several papers are highly relevant to ATF while explicitly noting that they do not use the term. QuickSilver presents “Contextual Token Fusion,” not “Adaptive Token Fusion” (Khanna et al., 27 Jun 2025). STTF is described as an ATF-style mechanism but not explicitly labeled that way (Tanvir et al., 23 Nov 2025). MoS is a close conceptual match only if ATF is interpreted broadly enough to include adaptive token-conditioned state routing rather than literal token merging (Liu et al., 15 Nov 2025). SEATS, finally, is best understood as fusion-aware adaptive pruning rather than token fusion proper, because its central operation is retention and removal rather than aggregation (Xin et al., 19 May 2026). This suggests that the boundary of ATF is still contested.

A second limitation is under-specification. STTF states that fusion is performed via learned gated averaging and that the threshold $m$05 is adapted per layer using a lightweight policy network trained with latency regularization, but it provides no formula or architecture for either component (Tanvir et al., 23 Nov 2025). TMCIR leaves open whether image-text matching is one-to-one, one-to-many, or many-to-many, and its printed loss equations contain indexing inconsistencies (Wang et al., 15 Apr 2025). Famba-V describes cosine-similarity-based matching and cross-layer scheduling heuristics but does not specify the exact matching algorithm beyond selecting the $m$06 most similar pairs (Shen et al., 2024). Such gaps make cross-paper ATF comparisons more difficult than headline results may suggest.

A third limitation is generalization and semantic safety. STTF reports severe overfitting: training accuracy approaches $m$07, while validation accuracy peaks near $m$08 and then plateaus, leading the authors to state that aggressive sparsity induction disrupts generalization unless paired with strong regularization (Tanvir et al., 23 Nov 2025). QuickSilver discusses “context bleed,” where locally similar tokens should remain distinct because of sentence-level role differences, and proposes contextual divergence filters and exclusion policies for named entities, punctuation, or sensitive terms (Khanna et al., 27 Jun 2025). TokenFusion depends heavily on reliable inter-modal alignment, whether same-position correspondence for homogeneous modalities or camera geometry for heterogeneous ones, so noisy alignment can turn adaptive substitution into systematic feature corruption (Wang et al., 2022).

A fourth limitation is that not all adaptive policies are fully learned or fully sample-conditional. ToFu’s layer schedule is fixed by depth even though token matching itself is content-dependent (Kim et al., 2023). Famba-V uses content-based pairing but static layer schedules and fixed merge budgets (Shen et al., 2024). Adaptive token merging for semantic communication searches threshold policies offline rather than learning them jointly with the backbone (Erak et al., 12 Sep 2025). SEATS uses backbone-specific heuristic layer boundaries for late removal of non-text tokens (Xin et al., 19 May 2026). A plausible implication is that future ATF systems may combine the robustness of training-free mechanisms with more explicit learned controllers or hardware-aware online policies.

The research trajectory already points toward such hybridization. STTF’s own future-work discussion proposes “Hybrid STTF+ANC Fusion,” combining token caching with adaptive branch routing (Tanvir et al., 23 Nov 2025). MoS identifies bidirectional routing as future work, which would extend adaptive fusion from understanding-to-generation injection to symmetric co-fusion (Liu et al., 15 Nov 2025). The semantic communication framework suggests adding hardware-aware objectives such as latency or energy, and possibly privacy as a fourth objective in multi-objective optimization (Erak et al., 12 Sep 2025). ATD shows that external prototype memory can drive both fusion and routing, indicating a path toward memory-augmented ATF with explicit category structure (Zhang et al., 3 Mar 2026). Together, these directions suggest that ATF is evolving from isolated token-merging heuristics toward broader adaptive-computation systems in which token fusion, routing, grouping, and retention are optimized jointly.