---
title: 'AdaSFFuse: Unified Multimodal Image Fusion'
url: https://www.emergentmind.com/topics/adasffuse
type: topic
---

# AdaSFFuse: Unified Multimodal Image Fusion

Searching arXiv for AdaSFFuse and closely related work to ground the article in current papers.
arxiv_search(query="AdaSFFuse multimodal image fusion", max_results=5)
arxiv_search(query="\"Task-Generalized Adaptive Cross-Domain Learning for Multimodal Image Fusion\"", max_results=10)
AdaSFFuse is a task-generalized multimodal image fusion (MMIF) framework introduced for a single unified architecture spanning Infrared-Visible Fusion (IVF), Multi-Focus Fusion (MFF), Multi-Exposure Fusion (MEF), and Medical Image Fusion (MIF). It is designed for aligned source images from different modalities, $\mathbf{I}_{m1}, \mathbf{I}_{m2} \in \mathbb{R}^{H \times W \times 1}$, and learns a fusion function $\mathbb{F}(\cdot)$ that produces a fused image $\mathbf{I}_{fuse} = \mathbb{F}(\mathbf{I}_{m1}, \mathbf{I}_{m2}; \omega_{\text{fuse}}) \in \mathbb{R}^{H \times W \times 1}$ integrating complementary structural detail, contrast, texture, and task-relevant semantics. The framework combines an Adaptive Approximate Wavelet Transform (AdaWAT) for learnable frequency decoupling with Spatial-Frequency Mamba Blocks for cross-domain fusion in spatial and frequency domains, and is trained jointly across heterogeneous imaging tasks under a unified loss formulation [2508.15505].

## 1. Problem formulation and scope

In the formulation used by AdaSFFuse, MMIF addresses the integration of complementary information from different sensing conditions or imaging modalities. The four tasks covered by the framework are IVF, MFF, MEF, and MIF. IVF fuses visible and infrared imagery, where VIS provides high spatial resolution and color while IR contributes thermal target saliency in low-light or night conditions. MFF fuses near-focus and far-focus images to obtain an image that is sharp everywhere. MEF combines under-exposed and over-exposed images to preserve detail in shadows and highlights. MIF fuses heterogeneous medical modalities such as CT-MRI, PET-MRI, and SPECT-MRI, where anatomical and functional information are distributed across inputs [2508.15505].

The framework is positioned against three persistent MMIF difficulties. The first is modality misalignment: different sensors induce distinct feature distributions and distinct frequency characteristics, so naïve spatial-domain fusion often fails to align them well. The second is high-frequency detail destruction: repeated downsampling and non-linear transformations in deep fusion networks can suppress edges, textures, and small structures. The third is task specificity: many earlier systems are tailored to one fusion task and rely on task-specific heuristics, limiting cross-task reuse. AdaSFFuse is explicitly framed as a cross-domain solution trained with shared parameters across tasks rather than as a collection of specialized pipelines [2508.15505].

A plausible implication is that the framework treats MMIF not primarily as a hand-crafted rule design problem, but as a representation-learning problem in which frequency decoupling, cross-domain alignment, and reconstruction are all jointly optimized.

## 2. Unified architecture and fusion pipeline

AdaSFFuse uses one encoder-AdaWAT-Mamba-AdaIWAT-decoder pipeline for all four tasks. Given two aligned grayscale inputs, each image is first passed through a shallow CNN encoder to produce feature maps
$$
\mathbf{F}_{m1}, \mathbf{F}_{m2} \in \mathbb{R}^{\frac{H}{2} \times \frac{W}{2} \times C},
$$
with spatial downsampling by $2$ and base channel count $C=64$. AdaWAT is then applied independently to each feature tensor, yielding four sub-bands per modality,
$$
\mathcal{F}_{LL_{m_i}}, \mathcal{F}_{LH_{m_i}}, \mathcal{F}_{HL_{m_i}}, \mathcal{F}_{HH_{m_i}} \in \mathbb{R}^{\frac{H}{4} \times \frac{W}{4} \times C},
$$
for $i \in \{1,2\}$, where $LL$ denotes low-frequency approximation and $LH$, $HL$, $HH$ denote high-frequency detail bands [2508.15505].

Band-wise fusion is then organized into low- and high-frequency streams. For high-frequency content,
$$
Hi_{fuse} = \sum_{n=1}^2 \{\mathcal{F}_{LH_{m_n}}, \mathcal{F}_{HL_{m_n}}, \mathcal{F}_{HH_{m_n}}\} \in \mathbb{R}^{\frac{H}{4} \times \frac{W}{4} \times 3C},
$$
while for low-frequency content,
$$
Lo_{fuse} = \sum_{n=1}^2 \mathcal{F}_{LL_{m_n}} \in \mathbb{R}^{\frac{H}{4} \times \frac{W}{4} \times C}.
$$
The paper notes that this “summation” corresponds to stacking or additive combination depending on implementation, with subsequent learnable fusion performed by Spatial-Frequency Mamba blocks [2508.15505].

Shallow fusion is performed through AdaD-SSD blocks, producing
$$
Hi^{s}_{fuse} \in \mathbb{R}^{\frac{H}{4} \times \frac{W}{4} \times 3C}, \qquad
Lo^{s}_{fuse} \in \mathbb{R}^{\frac{H}{4} \times \frac{W}{4} \times C}.
$$
These shallow fused bands are then combined through AdaIWAT and deeper Spatial-Frequency Mamba processing into a deep fused representation
$$
F^{d}_{fuse} \in \mathbb{R}^{\frac{H}{4} \times \frac{W}{4} \times C},
$$
which is up-sampled and decoded back to image space as $\mathbf{I}_{fuse} \in \mathbb{R}^{H \times W \times 1}$ [2508.15505].

No task-specific branches or hand-crafted fusion rules are introduced for IVF, MFF, MEF, or MIF. The same architecture is used across all tasks, with a unified loss and only task-specific intensity aggregation $M(\cdot)$ in the intensity term. This suggests that the model is intended to learn generic fusion priors—such as retaining strong gradients and structural similarity—rather than explicit task-conditioned rules [2508.15505].

## 3. Adaptive Approximate Wavelet Transform

AdaWAT is the frequency-decoupling component of AdaSFFuse. It is formulated as a learnable, adaptive approximation to wavelet decomposition rather than a fixed discrete wavelet transform. In its 1D presentation, for signal $x(t)$, scale $j$, translation $k$, wavelet $\psi_{j,k}$, and scaling function $\phi_{j,k}$,
$$
\psi_{j,k}(t) = 2^{j/2}\psi(2^{j}t - k), \quad \phi_{j,k}(t) = 2^{j/2}\phi(2^{j}t - k),
$$
and the decomposition at scale $j_0$ is written as
$$
x(t) = \sum_{j > j_0} \sum_k w_{j,k} \psi_{j,k}(t) + \sum_k v_{j_0,k} \phi_{j_0,k}(t),
$$
with coefficients
$$
w_{j,k} = \langle x(t), \psi_{j,k}(t) \rangle, \quad v_{j_0,k} = \langle x(t), \phi_{j_0,k}(t) \rangle.
$$
AdaWAT introduces adaptive analysis vectors
$$
\vec{u}_0[k] = 2^{1/2}\phi(2t - k), \quad \vec{u}_1[k] = 2^{1/2}\psi(2t - k),
$$
corresponding to low- and high-frequency filters, and uses the recursive convolution form
$$
\begin{aligned}
v_{j+1,p} &= \sum_k \vec{u}_0[k - 2p]\, a_{j,k}, \\
w_{j+1,p} &= \sum_k \vec{u}_1[k - 2p]\, a_{j,k}.
\end{aligned}
$$
[2508.15505]

For 2D feature maps, the method defines four convolution kernels:
$$
\begin{aligned}
K_{\mathcal{F}_{LL}} &= \vec{u}_0 \times \vec{u}_0^\top, \\
K_{\mathcal{F}_{LH}} &= \vec{u}_0 \times \vec{u}_1^\top, \\
K_{\mathcal{F}_{HL}} &= \vec{u}_1 \times \vec{u}_0^\top, \\
K_{\mathcal{F}_{HH}} &= \vec{u}_1 \times \vec{u}_1^\top,
\end{aligned}
$$
which produce the four frequency bands $\mathcal{F}_{LL}, \mathcal{F}_{LH}, \mathcal{F}_{HL}, \mathcal{F}_{HH} \in \mathbb{R}^{\frac{H}{4} \times \frac{W}{4} \times C}$. These kernels are implemented with grouped convolutions, allowing different channel groups to learn different wavelet-like parameters. The bases $\vec{u}_0$ and $\vec{u}_1$ are therefore not fixed analytic filters such as Haar or Daubechies, but are implicitly learned during training [2508.15505].

AdaWAT also applies band-specific dilated convolutions as a frequency enhancement stage. The low-frequency $LL$ band uses a $3 \times 3$ dilated convolution with dilation rate $r_{\text{DConv}} = 3$ to improve smoothness and global context, while the high-frequency $LH$, $HL$, and $HH$ bands use $3 \times 3$ dilated convolutions with $r_{\text{DConv}} = 1$ to preserve edges and textures. Reconstruction is handled by AdaIWAT, which uses similar wavelet kernels together with transpose convolutions with adaptive weights to recouple the four bands into $F^{d}_{fuse}$ with the stated goal of minimal information loss [2508.15505].

Relative to traditional DWT, the key distinction is adaptivity. Fixed analytical wavelets are task-agnostic and cannot adjust to IR-VIS, exposure, focus, or medical-domain statistics. AdaWAT replaces those fixed filters with jointly learned convolutional approximations, and the full decomposition-reconstruction loop is supervised indirectly by the overall fusion loss rather than by an explicit wavelet regularizer [2508.15505].

## 4. Spatial-Frequency Mamba and joint optimization

The second central component is the Spatial-Frequency Mamba block, denoted AdaD-SSD, which extends Mamba/Mamba2-style state-space modeling into a 2D spatial-frequency fusion setting. The starting point is the continuous-time linear state-space model
$$
\begin{aligned}
h_t &= A_t h_{t-1} + B_t x_t, \\
y_t &= C_t^\top h_t,
\end{aligned}
$$
where $x_t$ is input, $h_t$ is hidden state, and $A_t$, $B_t$, $C_t$ are learnable matrices. AdaD-SSD applies this logic to normalized fusion features $L_{in} \in \mathbb{R}^{H \times W \times C}$ by first projecting them to
$$
L' \in \mathbb{R}^{HW \times (2C' \times 2G \cdot d)},
$$
then reshaping for 2D processing [2508.15505].

A spatial-aware branch $\mathcal{S}(\cdot)$ applies a $3 \times 3$ convolution followed by SiLU,
$$
L_{\mathcal{S}} = \sigma\big(\mathbf{W}_{se}(L')\big),
$$
to strengthen local spatial dependencies. In parallel, a frequency filtering branch $\mathcal{T}(\cdot)$ applies FFT,
$$
\Lambda = \text{FFT}(L'),
$$
followed by a learned threshold $\lambda$ over the power spectrum and inverse FFT:
$$
L_{\mathcal{T}} = \text{IFFT}\big(\Lambda \odot (\|\Lambda\|^2 > \lambda)\big).
$$
The filtered spatial and frequency features are fused and split into
$$
\{\mathbf{X}, \mathbf{B}, \mathbf{C}, \mathbf{A}\} = \text{Split}(L_{\mathcal{S} \oplus L_{\mathcal{T}}}),
$$
with $\mathbf{X} \in \mathbb{R}^{HW \times C'}$, $\mathbf{B} \in \mathbb{R}^{HW \times G\cdot d}$, $\mathbf{C} \in \mathbb{R}^{HW \times G\cdot d}$, and $\mathbf{A} \in \mathbb{R}^{HW \times C'}$ [2508.15505].

The resulting 2D-SSD update is written for spatial coordinate $(i,j)$ and step $t$ as
$$
\begin{aligned}
H_{ij,t} &= A_t H_{ij,t - 1} + B_t X_{ij,t}, \\
Y_{ij,t} &= C_t^\top H_{ij,t},
\end{aligned}
$$
followed by gated activation, residual connection, and MLP to obtain $L_{out} \in \mathbb{R}^{H \times W \times C}$. The intended effect is simultaneous modeling of intra-region local structure and inter-region global dependencies, but with linear complexity in sequence length rather than quadratic attention cost [2508.15505].

Training is joint across the four MMIF tasks. The datasets specified are LLVIP for IVF with 12,025 training pairs and 3,463 test pairs; SICE for MEF with 542 training pairs and MEFB with 100 test pairs; Real-MFF with 710 pairs and MFI-WHU with 120 pairs for MFF training, evaluated on Lytro with 20 pairs and MFFW with 13 pairs; and Harvard medical datasets for MIF, comprising CT-MRI 160/24, PET-MRI 245/24, and SPECT-MRI 333/24 train/test splits, for a total of 738 training and 72 test pairs [2508.15505].

The overall loss is
$$
\mathcal{L}_{total} = \mu_1 \mathcal{L}_{ssim} + \mu_2 \mathcal{L}_{text} + \mu_3 \mathcal{L}_{int},
$$
with $\mu_1 = 10$ and $\mu_2 = \mu_3 = 20$. The SSIM term is
$$
\mathcal{L}_{ssim} = \sum_{i \in \{m1, m2\}} \left(1 - SSIM(\mathbf{I}_{fuse}, \mathbf{I}_i)\right),
$$
the texture term is
$$
\mathcal{L}_{text} = \frac{1}{HW} \left\| |\nabla \mathbf{I}_{fuse}| - \max\big(|\nabla \mathbf{I}_{m1}|, |\nabla \mathbf{I}_{m2}|\big) \right\|_1,
$$
and the intensity term is
$$
\mathcal{L}_{int} = \frac{1}{HW} \left\| \mathbf{I}_{fuse} - M(\mathbf{I}_{m1}, \mathbf{I}_{m2}) \right\|_1.
$$
Optimization uses Adam with learning rate $1 \times 10^{-4}$, batch size $120$ patches, patch size $128 \times 128$, and architectural depth parameters $N_1=2$, $N_2=4$, $C=64$ [2508.15505].

## 5. Empirical performance and ablation evidence

Evaluation uses EN, SD, SF, MI, SCD, VIF, Qabf, and SSIM. On IVF, AdaSFFuse is reported to achieve the best score on all listed metrics: EN $7.39$, SD $47.60$, SF $15.28$, MI $3.06$, SCD $1.48$, VIF $1.01$ as a tied highest value, Qabf $0.79$, and SSIM $1.51$. On MEF, it again records the best values on all listed metrics: EN $7.28$, SD $66.40$, SF $21.93$, MI $5.37$, SCD $0.76$, VIF $1.47$, Qabf $0.76$ as a tied best value, and SSIM $1.32$ [2508.15505].

On MFF, AdaSFFuse is reported as best or second best on all metrics, with EN $7.48$, SD $56.53$ as second to $56.59$, SF $20.44$, MI $4.87$, SCD $0.75$, VIF $1.26$, Qabf $0.72$ as second to $0.74$, and SSIM $1.78$. On MIF, it is likewise best or second best, with EN $5.07$, SD $79.96$, SF $30.81$, MI $2.49$ as second to $2.57$, SCD $1.59$, VIF $0.80$, Qabf $0.74$, and SSIM $1.49$ [2508.15505].

The ablation studies isolate the contribution of the two main innovations. For key components, the baseline gives EN $4.58$ and SSIM $0.89$; adding AdaWAT raises these to EN $6.47$ and SSIM $1.04$; adding shallow Mamba raises them to EN $7.25$ and SSIM $1.41$; and the full model with deep Mamba reaches EN $7.39$ and SSIM $1.51$. For frequency-decoupling methods, FFT gives SSIM $1.38$, Laplacian pyramid $1.33$, standard WAT $1.40$, and AdaWAT $1.51$. For AdaD-SSD design, SSM gives SSIM $1.38$, SSD $1.42$, 2D-SSD $1.47$, 2D-SSD with $\mathcal{S}$ remains $1.47$, and the full AdaD-SSD with $\mathcal{T}$ reaches $1.51$ [2508.15505].

The qualitative descriptions follow the same pattern. In IVF, the reported behavior is better foreground-background separation in low light and fused VIS-IR images with clear targets while preserving color through Y-channel fusion. In MEF, the description emphasizes balanced exposure, improved contrast, and preserved local details. In MFF, the method is described as preserving near- and far-focus details with less edge blur and better clarity. In MIF, CT-MRI and PET-MRI fusion are described as showing enhanced contrast and luminance, better separation of tissues and lesions, and more preserved fine structures [2508.15505].

## 6. Efficiency, downstream behavior, limitations, and naming

AdaSFFuse is presented as a compact network. The reported model size is $0.78$M parameters and approximately $82.81$G FLOPs at $512 \times 512$, compared with SwinFusion at approximately $254.92$G FLOPs and MambaDFuse at approximately $232.60$G FLOPs. Reported latency is $70.13$ ms, compared with $1157.83$ ms for SwinFusion and $227.97$ ms for MambaDFuse. These figures are consistent with the architectural choice to replace quadratic attention with state-space modeling while keeping the encoder-decoder backbone lightweight [2508.15505].

The paper also evaluates fused outputs in downstream tasks. On object detection using M3FD with YOLOv5 and YOLOv8, fused images are reported to yield higher AP/mAP than VIS and IR alone for most categories and in overall average; one explicit example is YOLOv5 mAP@[0.5:0.95] of $0.447$ for fusion versus $0.403$ for IR and $0.406$ for VIS. On semantic segmentation using MFNet and SegFormer, fusion images achieve the best average accuracy and IoU, reported as Acc $0.59$ and IoU $0.52$, with especially strong results for “Person,” “Curve,” and “Cone” [2508.15505].

The limitations are stated in terms of downstream-task mismatch and semantic selectivity. Enhanced visual fusion does not always improve downstream performance; some semantic cues can be diminished or blurred, and some small objects or fine-grained structures may be under-emphasized when the fusion objective favors global aesthetic quality or intensity consistency. Proposed future directions include task-aware end-to-end multimodal frameworks that optimize fusion jointly with downstream detection or segmentation, improved balance between modality-specific cues and fused semantics, unsupervised or self-supervised training, and extensions to video fusion or 3D medical data [2508.15505].

The official implementation is listed at `https://github.com/Zhen-yu-Liu/AdaSFFuse` [2508.15505]. AdaSFFuse should also be distinguished from the similarly named ADaFuSE, which is a different model for interactive text-to-image retrieval rather than multimodal image fusion [2603.21886].

Source: https://www.emergentmind.com/topics/adasffuse