AdaFuse: Adaptive Fusion Techniques for Heterogeneous Data

• AdaFuse is a family of adaptive fusion techniques that integrate heterogeneous data using context-sensitive, task-specific strategies across diverse domains.
• It employs methods like cross-attention, Fourier transforms, and Gumbel-Softmax sampling to optimize performance and reduce computational redundancy.
• Empirical results across medical imaging, pose estimation, video action recognition, and LLM decoding show significant gains in accuracy and efficiency despite computational challenges.

AdaFuse refers to a family of adaptive fusion techniques designed for application-specific integration of heterogeneous information in machine learning systems. Across domains, AdaFuse methods address fundamental challenges of multimodal fusion, reliability under uncertainty, and efficiency by adaptively selecting the optimal fusion strategy or units according to the context or data characteristics. Distinct instantiations of AdaFuse have been developed in medical image fusion, multiview human pose estimation, temporal action recognition, and LLM ensemble decoding, sharing the common principle of data-adaptive, context-sensitive fusion mechanisms (Gu et al., 2023, Zhang et al., 2020, Meng et al., 2021, Cui et al., 9 Jan 2026).

1. Adaptive Medical Image Fusion via Spatial-Frequential Cross Attention

In medical imaging, AdaFuse leverages encoder–decoder architectures with dual-branch spatial-frequential cross-attention to integrate features from multimodal inputs (e.g., CT/MRI/PET/SPECT). Each input image is processed by a shared-weight encoder extracting multiscale features. At each scale, a Spatial-Frequential Fusion (SFF) module fuses features through two components:

• Cross-Attention Fusion (CAF) Block: Fuses spatial features from each modality using Transformer encoders and cross-modal attention, yielding spatially fused representations. Queries and keys are exchanged across modalities; cross-attention weights modulate the relative contribution of each source feature at each patch.
• Fourier-Guided Fusion Branch (FGFB): Converts modality features into their log-magnitude Fourier spectra, fuses them using CAF, and recovers fused spatial features through inverse Fourier transform.

The outputs from spatial and frequential branches are further fused by an additional CAF, and the decoder reconstructs the final fused image via up-sampling and concatenation.

A novel loss function jointly optimizes content preservation (L2 distance to source averages) and structural fidelity (log-Frobenius norm gradient-tensor loss plus SSIM contrast loss):

$$\mathcal L = \lambda\,\mathcal L_{\rm content} + \mathcal L_{\rm structure}$$

with

$$\mathcal L_{\rm content} = \sum_{x,y}\left\|I_f(x,y) - \frac{I_1(x,y)+I_2(x,y)}{2}\right\|_2$$

and

$$\mathcal L_{\rm structure} = \log\left(1+\sum_{x,y}\|Z_{I_f}^{x,y} - Z_{I_c}^{x,y}\|_F^2\right) + \sum_i w_i(1-\mathrm{SSIM}(I_f,I_i))$$

AdaFuse achieves superior performance on standard benchmarks (CT–MRI: EN = 5.059, PSNR = 64.00 dB, MI = 3.357) and demonstrates consistent improvements across ablation studies. Limitations include the computation overhead of multi-scale Transformer blocks and lack of complex-phase modeling in the FGFB, pointing to lightweight attention and complex-domain network architectures as future avenues (Gu et al., 2023).

2. Adaptive Multiview Fusion for Human Pose Estimation

In multiview human pose estimation, AdaFuse is designed to mitigate view-dependent occlusion by learning view- and joint-specific fusion weights and employing heatmap sparsity and epipolar constraints. Each camera produces 2D joint heatmaps, which are extremely sparse due to the peakedness at joint locations. Rather than direct dense correspondence, AdaFuse restricts attention along epipolar lines, seeking high-confidence responses only where geometry is consistent.

For each joint, an adaptive weight $w_c^j$ is predicted for every view based on:

• Appearance Embedding: Encodes sharpness and peak statistics of the joint heatmap using a shallow CNN + fully connected layers.
• Geometry Embedding: Pools Sampson distances (epipolar consistency) across all other views, creating a vector of joint-wise reprojection metrics.

These are concatenated and mapped via an MLP, normalized by softmax, to obtain fusion weights. The fused heatmap for a joint $j$ in view $v$ is:

$$\hat{H}_v^j(\mathbf{x}) = w_v^j H_v^j(\mathbf{x}) + \sum_{u \neq v} w_u^j \max_{\mathbf{x}' \in \ell_{u\to v}(\mathbf{x})} H_u^j(\mathbf{x}')$$

End-to-end training minimizes a 2D heatmap MSE loss plus a 3D triangulated joint regression loss. Across standard datasets (Human3.6M, CMU Panoptic, TotalCapture, Occlusion-Person), AdaFuse improves MPJPE (Mean Per Joint Position Error), especially for occluded joints (e.g., 95.5% PCK@0.5 vs. 30.9% for single-view, mean MPJPE 12.6 mm vs. 48.1 mm). The approach generalizes to unseen camera configurations and domains without retraining, but lacks robustness when all views for a joint are occluded (Zhang et al., 2020).

3. Adaptive Temporal Fusion for Efficient Video Action Recognition

For temporal action recognition in video, AdaFuse introduces adaptive channel-wise temporal fusion to reduce redundancy and computational overhead in standard 2D CNNs. The core innovation lies in a policy network that, for each channel in a feature map at timestep $t$, determines—via Gumbel-Softmax stochastic sampling—whether to:

• Keep: Recompute the channel for the current frame.
• Reuse: Copy from the previous time step.
• Skip: Set the channel to zero, avoiding computation in this and downstream layers.

Formally, for channels $c$ at step $t$:

$$\tilde{F}_t^c = \mathbb{1}[p_t^c=0] F_t'^c + \mathbb{1}[p_t^c=1] H_{t-1}^c$$

with skips ( $p_t^c=2$ ) eliminating computation.

This channel-wise discrete policy is trained end-to-end via a Gumbel-Softmax estimator, with a composite loss of cross-entropy (recognition accuracy) and a FLOPs penalty. AdaFuse achieves ≈40% GFLOPs reduction versus TSN/TSM while maintaining or improving accuracy (e.g., AdaFuse-ResNet50 Top-1 = 41.9% at 22.1 GFLOPs vs. 18.7% at 32.9 GFLOPs for TSN). Channel reuse dominates at later layers, while skips are concentrated in early layers where redundancy is highest. Skipping alone degrades accuracy, but the full three-way adaptive policy achieves the optimal balance (Meng et al., 2021).

4. Adaptive Ensemble Decoding with Test-Time Scaling for LLMs

AdaFuse for LLM inference ensembles departs from fixed-granularity fusion by committing to adaptive word-level ensemble decisions during generation. The system proceeds in decoding rounds, with each round:

• Generating up to $M$ words per model using greedy decoding, provided a first-token probability margin (confidence gap) exceeds threshold $\tau_\Delta$.
• Under uncertainty (margin below threshold), triggering diversity-aware scaling: the model explores $B$ top-first-token word branches (maximizing diversity at this branching point), completing each branch to the nearest whitespace and collecting all candidates.

All candidate spans (from all models and branches) are rescored by computing the average token-normalized negative log-likelihood (NLL) over the ensemble:

$$F(\mathbf{s}^{(b)}) = \frac{1}{K} \sum_{k=1}^K \mathrm{NLL}_{m_k} (\mathbf{s}^{(b)})$$

where

$$\mathrm{NLL}_{m_k}(\mathbf{s}^{(b)}) = -\frac{1}{T_b} \sum_{t=1}^{T_b} \log p_{m_k}(z_t^{(b)} | \mathbf{y}, z_{<t}^{(b)})$$

The span with the lowest average NLL is appended to the sequence.

Across NaturalQuestions, SQuAD, TriviaQA, GSM8K, and flores translation benchmarks, AdaFuse yields a mean 6.88% relative improvement over fixed-granularity ensemble baselines, with notable gains in open-domain QA (up to +10.25 points EM on SQuAD) and arithmetic reasoning (GSM8K accuracy 90.25 vs. 81.05 single-model). Adaptive commitment strategies outperform fixed-length word commitments and token-level beam search, while moderate scaling factors ( $B=3$ ) are sufficient for most gains (Cui et al., 9 Jan 2026).

5. Comparative Summary and Cross-Domain Insights

While differing in domain and architecture, all AdaFuse approaches share an adaptive, context-sensitive fusion paradigm that produces consistent accuracy or efficiency improvements over non-adaptive baselines. Table 1 summarizes the design axes across four canonical AdaFuse systems:

Domain	Adaptive Unit	Gating/Policy Signal	Empirical Gains
Medical image fusion	Spatial/frequency	Cross-attention + Fourier analysis	+12.9% EN, +0.43 FMI (CT–MRI)
Multiview pose	View/joint heatmaps	Appearance+geometry via MLP	–3 mm MPJPE (Panoptic), +64% PCK (occl)
Temporal action recog.	Channel/time	Channel policy network	~40% FLOPs ↓, same/higher accuracy
LLM ensemble decoding	Word/candidate span	Uncertainty margin, diversity	+6.88% rel. across QA/MT/AR

Common limitations include increased computational cost of adaptive policies, the challenge of model selection and calibration under shifts, and the need to engineer suitable gating or policy signals for the target domain. Future progress is likely in lightweight attention, phase-aware spectral fusion, improved self-supervision for fusion under data scarcity, higher-level commitment strategies in LLM ensembling, and temporal fusion for pose estimation.