FS-Diff: Joint Fusion & Super-Resolution

• FS-Diff is a semantic guidance and clarity-aware method that formulates multimodal image fusion and super-resolution as a conditional diffusion generation problem.
• It uses a modified U-Net with cross-attention, integrated CA-CLIP clarity sensing, and bidirectional feature Mamba to extract and merge global cross-modal features.
• Empirical evaluations show significant improvements in VIF, SSIM, and downstream tasks, outperforming sequential fusion and super-resolution pipelines on various benchmarks.

FS-Diff is a semantic guidance and clarity-aware joint image fusion and super-resolution method that formulates simultaneous multimodal fusion and super-resolution as a conditional generation problem. It is designed for settings in which multimodal inputs such as visible and infrared images exhibit low resolution, corrupted target and background structures, and weak semantic information, conditions under which sequential pipelines can produce suboptimal results. The method initializes the desired fused result as pure Gaussian noise, conditions denoising on the source images and a semantic guidance vector, uses bidirectional feature Mamba to extract global cross-modal features, and employs a modified U-Net to generate high-resolution fused images with cross-modal features and abundant semantic information (Jie et al., 11 Sep 2025).

1. Problem formulation and task scope

FS-Diff casts simultaneous multimodal fusion and super-resolution as sampling a high-resolution fused image $\mathbf{F}_{\mathrm{fused}}$ from a learned conditional distribution

$$p(\mathbf{F}_{\mathrm{fused}} \mid \mathbf{x},\mathbf{y}),$$

where $\mathbf{x}\in\mathbb{R}^{3\times H\times W}$ is the low-res visible input and $\mathbf{y}\in\mathbb{R}^{1\times H\times W}$ is the low-res infrared input. A diffusion model gradually refines a pure Gaussian noise state $\mathbf{F}_T\sim\mathcal{N}(0,I)$ back to $\mathbf{F}_0=\mathbf{F}_{\mathrm{fused}}$ in $T$ denoising steps, and at each step the reverse kernel is conditioned on $\mathbf{x}$, $\mathbf{y}$, and a semantic guidance vector $e^{ci}$ (Jie et al., 11 Sep 2025).

This formulation integrates fusion and up-sampling inside the diffusion network $$p(\mathbf{F}_{\mathrm{fused}} \mid \mathbf{x},\mathbf{y}),$$0 rather than decomposing them into separate stages. The paper states that this avoids error-amplification from sequential SR $$p(\mathbf{F}_{\mathrm{fused}} \mid \mathbf{x},\mathbf{y}),$$1 fusion pipelines. A plausible implication is that the model is intended to preserve both modality-complementarity and scale-dependent detail within a single generative trajectory, rather than treating super-resolution as a preprocessing or postprocessing operation.

The method is positioned against real-world multimodal scenarios such as military reconnaissance and long-range detection missions, where the source images may be both semantically weak and degraded. In this setting, FS-Diff treats low-level restoration and cross-modal information fusion as a single conditional synthesis problem rather than as disjoint objectives.

2. Clarity sensing and semantic guidance

A defining component of FS-Diff is its clarity sensing and semantic guidance mechanism, denoted CLSE and implemented with CA-CLIP. Its stated goal is to decide whether each input, visible or infrared, is “clear” or “blurred,” and then extract the best semantic embedding from a pretrained CLIP (Jie et al., 11 Sep 2025).

CA-CLIP produces three outputs per image: content embeddings $$p(\mathbf{F}_{\mathrm{fused}} \mid \mathbf{x},\mathbf{y}),$$2 and resolution-type logits $$p(\mathbf{F}_{\mathrm{fused}} \mid \mathbf{x},\mathbf{y}),$$3. It also uses text embeddings for the words “clear” and “blur,” denoted $$p(\mathbf{F}_{\mathrm{fused}} \mid \mathbf{x},\mathbf{y}),$$4 and $$p(\mathbf{F}_{\mathrm{fused}} \mid \mathbf{x},\mathbf{y}),$$5. The visibility and infrared clarity decisions are then defined through cosine-similarity comparisons: $$p(\mathbf{F}_{\mathrm{fused}} \mid \mathbf{x},\mathbf{y}),$$6 If only the visible image is clear, $$p(\mathbf{F}_{\mathrm{fused}} \mid \mathbf{x},\mathbf{y}),$$7; if only the infrared image is clear, $$p(\mathbf{F}_{\mathrm{fused}} \mid \mathbf{x},\mathbf{y}),$$8; if both are blurred, the semantic guidance is chosen by

$$p(\mathbf{F}_{\mathrm{fused}} \mid \mathbf{x},\mathbf{y}),$$9

with the maximum taken element-wise.

During diffusion, $\mathbf{x}\in\mathbb{R}^{3\times H\times W}$0 enters $\mathbf{x}\in\mathbb{R}^{3\times H\times W}$1 via cross-attention. At a given U-Net layer, queries come from the fused-feature tensor $\mathbf{x}\in\mathbb{R}^{3\times H\times W}$2, while keys and values are obtained from the linearly projected semantic embedding. This makes the semantic signal conditional not only on the input modalities but also on their assessed clarity state.

The paper further notes that CA-CLIP is pretrained via a clarity-aware contrastive loss, as in DA-CLIP [Luo et al. ’23], to predict “clear” versus “blur” while aligning image and text embeddings. That pretraining step can be performed offline, whereas during diffusion training both $\mathbf{x}\in\mathbb{R}^{3\times H\times W}$3 and the small control-branch in CA-CLIP, termed the “image controller,” are optimized jointly under the same mean-squared noise-prediction objective.

3. Bidirectional Feature Mamba and cross-modal feature construction

FS-Diff uses bidirectional feature Mamba, abbreviated BFM, to extract global features of the multimodal inputs. At denoising step $\mathbf{x}\in\mathbb{R}^{3\times H\times W}$4, the model forms

$\mathbf{x}\in\mathbb{R}^{3\times H\times W}$5

then divides $\mathbf{x}\in\mathbb{R}^{3\times H\times W}$6 into non-overlapping $\mathbf{x}\in\mathbb{R}^{3\times H\times W}$7 patches to obtain $\mathbf{x}\in\mathbb{R}^{3\times H\times W}$8 with $\mathbf{x}\in\mathbb{R}^{3\times H\times W}$9 (Jie et al., 11 Sep 2025).

Each patch is linearly projected and prepended with a class token: $\mathbf{y}\in\mathbb{R}^{1\times H\times W}$0 The token sequence is then processed through $\mathbf{y}\in\mathbb{R}^{1\times H\times W}$1 Vision-Mamba blocks using residual propagation,

$\mathbf{y}\in\mathbb{R}^{1\times H\times W}$2

The paper characterizes each Vision-Mamba block as a bidirectional State-Space Model that implements a 1-D convolution on hidden states in both forward and backward directions, fuses them by a learned gate, and projects back to tokens. After $\mathbf{y}\in\mathbb{R}^{1\times H\times W}$3 layers, the sequence is un-patchified to recover

$\mathbf{y}\in\mathbb{R}^{1\times H\times W}$4

which becomes the U-Net’s feature conditioning input.

This design gives FS-Diff a distinct conditioning pathway: the diffusion process is not driven solely by the noisy fused image and timestep, but by a tokenized representation of the two source modalities together with the current fused-state estimate. The explicit use of a bidirectional SSM indicates that long-range spatial dependencies are handled in a global token space before denoising proceeds in the convolutional backbone.

4. Diffusion process, denoiser architecture, and optimization

The forward noising process in FS-Diff is

$\mathbf{y}\in\mathbb{R}^{1\times H\times W}$5

with $\mathbf{y}\in\mathbb{R}^{1\times H\times W}$6, and the marginal is

$\mathbf{y}\in\mathbb{R}^{1\times H\times W}$7

Training samples a random timestep $\mathbf{y}\in\mathbb{R}^{1\times H\times W}$8 and Gaussian noise $\mathbf{y}\in\mathbb{R}^{1\times H\times W}$9, then constructs

$\mathbf{F}_T\sim\mathcal{N}(0,I)$0

The denoiser $\mathbf{F}_T\sim\mathcal{N}(0,I)$1 is trained to predict $\mathbf{F}_T\sim\mathcal{N}(0,I)$2 by

$\mathbf{F}_T\sim\mathcal{N}(0,I)$3

(Jie et al., 11 Sep 2025).

The denoiser is a modified U-Net. It accepts $\mathbf{F}_T\sim\mathcal{N}(0,I)$4 through feature-wise affine layers, injects the CLIP-derived semantic guidance through cross-attention, and incorporates skip-connections from $\mathbf{F}_T\sim\mathcal{N}(0,I)$5. The source images and semantics therefore enter both through explicit conditioning and through the cross-modal feature path produced by BFM.

The reverse denoising step is parameterized as

$\mathbf{F}_T\sim\mathcal{N}(0,I)$6

with

$\mathbf{F}_T\sim\mathcal{N}(0,I)$7

Algorithm 2 in the paper gives the full inference loop from $\mathbf{F}_T\sim\mathcal{N}(0,I)$8 to $\mathbf{F}_T\sim\mathcal{N}(0,I)$9.

Taken together, these components define an end-to-end conditional diffusion architecture in which semantic clarity assessment, global multimodal feature extraction, and super-resolving fusion are optimized within one generative model.

5. Data regime, evaluation protocol, and benchmark structure

The experimental scope spans multimodal visible–infrared fusion, medical image fusion, and multifocus fusion. The datasets are organized as follows (Jie et al., 11 Sep 2025):

Regime	Datasets	Notes
VIRF	LLVIP, M³FD, MSRS, AVMS	AVMS is the paper’s aerial view multiscene benchmark
MIF	Harvard MRI–PET, MRI–SPECT	Medical multimodal fusion
Multifocus	Lytro, MFI	Multifocus fusion

Training simulates low-resolution inputs by bicubic downsampling high-resolution source images. High-resolution fusion targets are produced by CDD, described as a fusion-only AE. Testing uses the standard “crop→downsample→fuse” protocol on held-out sets such as LLVIP-val.

The evaluation metrics include image-fidelity and structural measures—VIF, SSIM, PSNR, and $\mathbf{F}_0=\mathbf{F}_{\mathrm{fused}}$0—together with LPIPS for perceptual quality and MSE for error. The paper also extends evaluation to high-level downstream tasks: YOLOv8 detection measured by mAP@[.5,.75,.5:.95] and BiSeNet segmentation measured by mIoU. Qualitative comparisons are reported at $\mathbf{F}_0=\mathbf{F}_{\mathrm{fused}}$1, $\mathbf{F}_0=\mathbf{F}_{\mathrm{fused}}$2, and $\mathbf{F}_0=\mathbf{F}_{\mathrm{fused}}$3 scales under single and dual blur, with region-of-interest zoom-ins for target shapes and textures.

One detail requires careful reading. The abstract describes AVMS as “covering 600 pairs of images,” whereas the detailed summary lists AVMS as “859 aligned VI/IR pairs, 3821 detections in 7–8 scenes at different lighting.” The paper materials as presented therefore contain two different dataset-size descriptions. This suggests that readers consulting the implementation or supplementary materials should verify the exact benchmark split and counting convention.

6. Empirical findings, ablations, and computational profile

The paper reports that across VIS+IR datasets at $\mathbf{F}_0=\mathbf{F}_{\mathrm{fused}}$4, FS-Diff is consistently top-1 on VIF/$\mathbf{F}_0=\mathbf{F}_{\mathrm{fused}}$5/SSIM/LPIPS on MSRS, M³FD, LLVIP, and AVMS, often beating the second-place method by 3–5 percentage points. On medical datasets, specifically MRI–PET and MRI–SPECT at $\mathbf{F}_0=\mathbf{F}_{\mathrm{fused}}$6, it matches or exceeds the best result in VIF/$\mathbf{F}_0=\mathbf{F}_{\mathrm{fused}}$7/SSIM and recovers semantic and structural edges better than DDFM and U2Fusion. On multifocus data at $\mathbf{F}_0=\mathbf{F}_{\mathrm{fused}}$8, it achieves the highest VIF and SSIM and the lowest LPIPS on Lytro and MFI, with the summary reporting VIF $\mathbf{F}_0=\mathbf{F}_{\mathrm{fused}}$9 versus $T$0, SSIM $T$1 versus $T$2, and LPIPS $T$3 versus $T$4 (Jie et al., 11 Sep 2025).

The downstream-task evaluations are also explicit. Detection [email protected] rises from approximately $T$5 for the best baselines to $T$6 on LLVIP and $T$7 on AVMS. Segmentation mIoU rises from approximately $T$8 to $T$9 on MSRS and from $\mathbf{x}$0 to $\mathbf{x}$1 on AVMS. These results are presented as evidence that the fused super-resolved outputs preserve task-relevant semantics rather than merely improving low-level perceptual scores.

Ablation results attribute the improvement to both major architectural components. The paper states that BFM and CLSE each provide approximately $\mathbf{x}$2–$\mathbf{x}$3 relative gain, and that combining them yields the full FS-Diff improvement. This suggests that the method’s performance is not reducible to either semantic guidance alone or cross-modal token processing alone; rather, the reported gain depends on the interaction between clarity-aware semantics and global multimodal feature extraction.

The reported complexity is $\mathbf{x}$4 M parameters and approximately $\mathbf{x}$5 k G-FLOPs for $\mathbf{x}$6 input. The model is characterized as heavier than pure fusion networks but practical relative to fusion-plus-super-resolution pipelines. The same summary concludes that FS-Diff is the first end-to-end diffusion-based network that simultaneously upsamples and fuses multimodal inputs, guided by semantic clarity awareness and a bidirectional global feature extractor.

7. Relation to similarly named diffusion methods

The name “FS-Diff” can be confused with several distinct diffusion-based systems, but these works address different tasks and use different guidance mechanisms.

“FSDiffReg: Feature-wise and Score-wise Diffusion-guided Unsupervised Deformable Image Registration for Cardiac Images” defines an unsupervised deformable registration framework for 3D cardiac MR, using a pretrained denoising diffusion model as a semantic and topological guide through a Feature-wise Diffusion-Guided module and a Score-wise Diffusion-Guided module (Qin et al., 2023). Its objective is voxelwise deformation-field estimation, not multimodal image fusion and super-resolution.

“DifFSS: Diffusion Model for Few-Shot Semantic Segmentation” introduces a few-shot semantic segmentation paradigm in which a frozen conditional diffusion model based on ControlNet and Stable Diffusion generates diverse auxiliary support images from masks, HED boundaries, or scribbles (Tan et al., 2023). Its role for diffusion is support-set augmentation rather than conditional denoising of a fused high-resolution target.

“FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion LLMs” addresses long-text generation by a discrete flow-matching model that exposes the step budget directly to the network and uses a cumulative scalar together with RK-4 shortcut distillation (Monsefi et al., 24 Sep 2025). Despite the superficial acronymic similarity, it belongs to diffusion language modeling rather than computer vision.

Accordingly, FS-Diff denotes, in the strict sense established by its title and formulation, a semantic guidance and clarity-aware simultaneous multimodal image fusion and super-resolution method. The shared use of diffusion models across these works does not imply methodological interchangeability; the commonality is the generative prior, whereas the conditioning pathways, objectives, and application domains differ substantially.