---
title: 'SGDFuse: SAM-Guided IV Fusion'
url: https://www.emergentmind.com/topics/sgdfuse
type: topic
---

# SGDFuse: SAM-Guided IV Fusion

Searching arXiv for the specific term and closely related fusion papers to ground the article in current literature.
SGDFuse, introduced in "SAM-Guided Diffusion for High-Fidelity Infrared and Visible Image Fusion" [2508.05264], is an infrared-visible image fusion (IVIF) framework that combines thermal radiation information from infrared images with the texture and structural detail of visible images through a two-stage pipeline. Its defining premise is that IVIF is not only a low-level signal-combination problem but also a semantic and generative one: a fused image should preserve infrared salient targets, retain visible texture and structural detail, maintain global visual fidelity without blur or artifacts, and remain useful for downstream tasks such as object detection and semantic segmentation. To address what the paper describes as a “semantic blindness” problem in prior methods, SGDFuse uses semantic masks from the Segment Anything Model (SAM) as explicit priors and refines a preliminary fused image with a conditional diffusion model [2508.05264].

## 1. Problem setting and conceptual basis

SGDFuse is formulated for IVIF, where the objective is to combine two complementary modalities. Infrared images provide strong thermal radiation cues and are particularly useful for highlighting salient targets in low-light, smoke, or night scenes, whereas visible images provide rich texture, edge, and color or structural information [2508.05264]. The method is motivated by two failure modes identified in existing fusion systems. First, methods based primarily on low-level intensity, gradient, or local contrast often preserve pixels without explicitly distinguishing target from background, which can blur boundaries, suppress thermal targets, or lose fine structures. Second, fusion is treated as an image generation problem: insufficiently generative methods can introduce oversmoothing, distortions, or structural inconsistencies.

A central claim of the method is that semantic understanding should be explicit rather than implicit. SGDFuse therefore treats SAM-generated masks as direct scene priors and uses a diffusion process for semantic-guided refinement. This design suggests that semantic localization and generative restoration are jointly necessary when the goal is not merely visual sharpness but semantically coherent fused imagery that remains effective for downstream vision tasks [2508.05264].

## 2. Two-stage framework and data flow

The architecture is organized as a two-stage process. Stage I performs preliminary multi-modal feature fusion, extracting modality-specific features from infrared and visible images and producing an initial fused RGB image \(F_1 \in \mathbb{R}^{H\times W\times 3}\). Stage II then performs SAM-guided conditional diffusion with coarse-to-fine denoising, using semantic masks from both modalities together with \(F_1\) to refine boundaries, detail, and semantic consistency [2508.05264].

| Stage | Main components | Output |
|---|---|---|
| Stage I | MSFEM branch for infrared, Transformer block for visible, cross-attention fusion | Preliminary fused image \(F_1\) |
| Stage II | SAM masks, conditional diffusion model, U-Net denoiser, HFAH | Final fused image \(I_f\) |

The key conditioning tensor for Stage II is formed by concatenating the preliminary fused image with two SAM masks:
\[
I_0 = Concat(F_1, M_{ir}, M_{vis}) \in \mathbb{R}^{H\times W\times 5}.
\]
Here \(M_{ir} \in \mathbb{R}^{H\times W\times 1}\) and \(M_{vis} \in \mathbb{R}^{H\times W\times 1}\) are the SAM-generated masks for infrared and visible images, respectively. The paper attributes three semantic advantages to this construction: target preservation, boundary fidelity, and semantic awareness. In ablation, removing SAM significantly degrades all metrics, which the authors treat as evidence that these masks function as central semantic priors rather than as auxiliary inputs [2508.05264].

## 3. Stage I: preliminary multi-modal feature fusion

Stage I comprises two modality-specific branches. The infrared branch uses a Multi-Scale Feature Enhancement Module (MSFEM), while the visible branch uses a Transformer block with multi-head self-attention to capture global context, fine-grained texture, and long-range dependencies [2508.05264].

For the infrared stream, the feature map \(F_{ir}\) is processed with four convolutional branches:
\[
F_i = Conv_{j\times j}(F_{ir}), \quad i=1,2,3,4;\; j=1,3,5,7.
\]
This creates multi-scale features with receptive fields spanning shallow detail and broader structural context. The larger-scale branches are concatenated,
\[
F_{ms} = Concat(F_2, F_3, F_4),
\]
then enhanced by depthwise and pointwise convolutions,
\[
F_{enh} = DWConv_{3\times3}\big(Conv_{1\times1}(DWConv_{3\times3}(F_{ms}))\big).
\]
The shallow branch is preserved separately and concatenated with the enhanced feature:
\[
F_{cat} = Concat(F_{enh}, F_1).
\]
A \(1\times1\) convolution followed by a sigmoid produces normalized weights, and a residual connection yields the final output:
\[
F_{out} = F_{ir} + \sigma(Conv_{1\times1}(F_{cat})).
\]
The paper interprets this as robust multi-scale structure modeling that preserves original infrared information while enhancing salient thermal regions. It further reports that stacking MSFEM three times gives the best tradeoff [2508.05264].

The visible branch uses a Transformer block rather than an explicitly convolutional design. The text does not give a formal equation for this block, but states that it is used to extract global context, fine-grained texture, and long-range dependencies from the visible image. Cross-modal fusion is then performed by dynamically aligning and selectively fusing the extracted infrared and visible features through cross-attention interaction, generating the preliminary fused image \(F_1\). This initial output is designed to preserve salient infrared targets and visible detail, but it is not the final reconstruction; its principal role is to provide a structured initialization for the semantic refinement stage [2508.05264].

## 4. Stage II: SAM-guided conditional diffusion and semantic refinement

Stage II is the method’s distinctive contribution. It uses a DDPM-style conditional diffusion process to refine the preliminary fusion under explicit semantic guidance from SAM masks [2508.05264]. The forward diffusion process gradually adds Gaussian noise to the conditional input \(I_0\):
\[
P(I_t|I_{t-1}) = \mathcal{N}(I_t;\sqrt{\alpha_t}I_{t-1},(1-\alpha_t)\mathbf{I}),
\]
with closed-form sampling
\[
I_t=\sqrt{\bar{\alpha}_t}I_0+\sqrt{1-\bar{\alpha}_t}\varepsilon,\quad\varepsilon\sim\mathcal{N}(0,\mathbf{I}),
\]
where
\[
\bar{\alpha}_t=\prod_{i=1}^{t}\alpha_i.
\]

The reverse process reconstructs the signal by predicting noise and denoising iteratively:
\[
Q(I_{t-1}|I_t)=\mathcal{N}(I_{t-1};\mu_\theta(I_t,t),\sigma_t^2\mathbf{I}),
\]
with
\[
\sigma_t^2=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t}\cdot\beta_t, \qquad \beta_t = 1-\alpha_t,
\]
and mean
\[
\mu_\theta(I_t,t)=\frac{1}{\sqrt{\alpha_t}}\left(I_t-\frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\,\varepsilon_\theta(I_t,t)\right).
\]
The paper describes this as coarse-to-fine denoising: early steps remove coarse noise and shape global structure, while later steps refine boundaries and local details.

Training uses the simplified DDPM noise-prediction objective
\[
\mathcal{L}_{\mathrm{diff}}=\left\|\gamma-\varepsilon_\theta\left(\sqrt{\bar{\alpha}_t}I_0+\sqrt{1-\bar{\alpha}_t}\gamma,t\right)\right\|_2^2,
\]
where \(\gamma \sim \mathcal{N}(0,\mathbf{I})\). The denoising network \(\varepsilon_\theta(\cdot,\cdot)\) is a U-Net inspired by SR3, with a contracting path of 5 convolutional layers, an expanding path of 5 corresponding convolutional layers, and a diffusion head that predicts noise with the same dimensionality as the input. Because the input has five channels, the denoiser jointly models fused visual appearance, infrared semantics, and visible semantics. The decoder is augmented with a Hierarchical Feature Aggregation Head (HFAH), which aggregates multi-level decoded features, uses spatial attention, strengthens edge, structure, and semantic region modeling, and improves boundary fidelity and detail recovery. The final fused output is
\[
I_f \in \mathbb{R}^{H\times W\times 3},
\]
and the final fusion head uses multiple \(3\times3\) convolutions followed by a Tanh activation [2508.05264].

## 5. Losses, supervision, and evaluation protocol

SGDFuse uses separate losses for the two stages. For Stage I, the paper defines a gradient loss
\[
L_{grad}^{1}=\frac{1}{HW}\parallel\nabla F_{1}-\nabla I_{vis}\parallel
\]
and an intensity loss
\[
L_{int}^{1}=\frac{1}{HW}\parallel F_{1}-I_{ir}\parallel.
\]
The manuscript writes the stage-I total loss as
\[
L_{stage1}=L_{int}^{1}+L_{grad}^{2}.
\]
The accompanying explanation states that this appears to be a typo; based on the preceding definitions, the intended combination is the infrared intensity consistency term and the visible gradient consistency term [2508.05264].

Stage II uses mask-guided regional supervision. A joint semantic mask is constructed as
\[
M=\max(M_{ir},M_{vis}),
\]
followed by a mask-guided intensity loss
\[
L_{int}^{mask}=\frac{1}{HW}\parallel M\cdot(I_f-\max(I_{ir},I_{vis}))\parallel_{1},
\]
and a mask-guided gradient loss
\[
L_{grad}^{mask}=\frac{1}{HW}\parallel M\cdot(\nabla I_f-\max(\nabla I_{ir},\nabla I_{vis}))\parallel_{1}.
\]
The stage-II total loss is
\[
L_{stage2}=\lambda_{1}\cdot L_{int}^{mask}+\lambda_{2}\cdot L_{grad}^{mask},
\]
with best weights \(\lambda_1 = 1.5\) and \(\lambda_2 = 1\). These losses are region-weighted rather than uniform across pixels, which is consistent with the framework’s emphasis on semantically important regions [2508.05264].

Training is conducted in PyTorch on an NVIDIA RTX 4090 using the MSRS dataset, with 1,083 visible-infrared training pairs and 361 test pairs, patch size \(160\times160\), Adam optimization, learning rate \(1\times10^{-4}\), batch size 24, and 200 epochs. The model is trained in a two-stage manner: Stage I learns the preliminary fusion, and Stage II learns diffusion-based semantic refinement. At inference, images are processed at original resolution and the model outputs a 3-channel fused image [2508.05264].

Evaluation is reported on four public IVIF datasets: MSRS, M\(^3\)FD, LLVIP, and RoadScene. The paper uses seven image-fusion metrics: EN, SD, SF, MI, SCD, VIF, and \(Q^{abf}\), with higher values reported as better in the tables. Downstream-task adaptability is evaluated with YOLOv5 for object detection and DeepLabV3+ for semantic segmentation [2508.05264].

## 6. Empirical behavior, ablations, limitations, and nomenclature

The reported experimental results show consistent top-ranked or near-top-ranked performance across datasets. On MSRS, SGDFuse achieves EN \(6.81\), SD \(45.28\), SF \(13.27\), MI \(2.99\), SCD \(1.73\), VIF \(1.08\), and \(Q^{abf}\) \(0.74\), with best performance on EN, SF, SCD, VIF, and \(Q^{abf}\). On M\(^3\)FD, it is reported as top-ranked on all listed metrics, including EN \(6.99\), SF \(13.27\), MI \(2.99\), SCD \(1.69\), VIF \(1.01\), and \(Q^{abf}\) \(0.69\). On LLVIP, it achieves EN \(7.12\), SD \(43.35\), SF \(16.15\), MI \(2.79\), SCD \(1.65\), VIF \(0.99\), and \(Q^{abf}\) \(0.71\), with best scores on almost all metrics. On RoadScene, it performs best on MI \(2.89\), SCD \(1.84\), VIF \(0.82\), and \(Q^{abf}\) \(0.62\) [2508.05264].

The ablation studies attribute the largest degradation to removing SAM. Additional variants remove only the infrared or visible SAM mask, and the results are described as showing that dual-modality semantic masks are complementary. Removing Stage I yields insufficient structural priors and blur or distortion; removing Stage II yields weaker semantic consistency and fine detail; removing diffusion lowers all metrics and degrades clarity, structural integrity, artifact reduction, and high-fidelity reconstruction; removing HFAH worsens boundary fidelity and texture detail. The number of stacked MSFEM and Transformer blocks was varied from 2 to 4, and performance improved up to 3 layers before saturating or slightly declining, leading to a final design with three MSFEM and three Transformer blocks [2508.05264].

The downstream-task results are presented as evidence that the fused images are semantically useful rather than only visually plausible. With YOLOv5 on MSRS, SGDFuse improves detection performance over competing fused-image methods, especially for Person and Car. With DeepLabV3+, it reports the best semantic-segmentation mIoU of \(0.819\) across Background, Car, Person, Curve, Color Cone, and Bike. This suggests that semantic-aware fusion can benefit recognition pipelines more directly than methods optimized only for appearance [2508.05264].

A stated future direction is real-time optimization, indicating that faster inference remains an open engineering objective. A plausible implication is that performance may depend on mask reliability, because the framework conditions its refinement stage on SAM-generated semantic masks. The method should also be distinguished from similarly named fusion approaches in other domains: Sparse Dense Fusion (SDF) addresses camera-LiDAR fusion for 3D object detection [2304.04179], and GS-Fuse addresses event-driven financial forecasting through Granger-supervised gated fusion and multi-granularity alignment [2605.28520]. In the present usage, SGDFuse specifically denotes a SAM-guided diffusion framework for high-fidelity infrared-visible image fusion [2508.05264].

Source: https://www.emergentmind.com/topics/sgdfuse