SGDFuse: SAM-Guided IV Fusion

• SGDFuse is an IVIF framework that fuses thermal data with visible textures by combining multi-scale feature extraction and SAM-guided conditional diffusion.
• The method first extracts modality-specific features from infrared and visible images, then refines a preliminary fused output using semantic masks for enhanced detail and target preservation.
• Extensive evaluations show that SGDFuse improves visual fidelity and boosts downstream tasks like object detection and semantic segmentation.

Searching arXiv for the specific term and closely related fusion papers to ground the article in current literature. SGDFuse, introduced in "SAM-Guided Diffusion for High-Fidelity Infrared and Visible Image Fusion" (Zhang et al., 7 Aug 2025), is an infrared-visible image fusion (IVIF) framework that combines thermal radiation information from infrared images with the texture and structural detail of visible images through a two-stage pipeline. Its defining premise is that IVIF is not only a low-level signal-combination problem but also a semantic and generative one: a fused image should preserve infrared salient targets, retain visible texture and structural detail, maintain global visual fidelity without blur or artifacts, and remain useful for downstream tasks such as object detection and semantic segmentation. To address what the paper describes as a “semantic blindness” problem in prior methods, SGDFuse uses semantic masks from the Segment Anything Model (SAM) as explicit priors and refines a preliminary fused image with a conditional diffusion model (Zhang et al., 7 Aug 2025).

1. Problem setting and conceptual basis

SGDFuse is formulated for IVIF, where the objective is to combine two complementary modalities. Infrared images provide strong thermal radiation cues and are particularly useful for highlighting salient targets in low-light, smoke, or night scenes, whereas visible images provide rich texture, edge, and color or structural information (Zhang et al., 7 Aug 2025). The method is motivated by two failure modes identified in existing fusion systems. First, methods based primarily on low-level intensity, gradient, or local contrast often preserve pixels without explicitly distinguishing target from background, which can blur boundaries, suppress thermal targets, or lose fine structures. Second, fusion is treated as an image generation problem: insufficiently generative methods can introduce oversmoothing, distortions, or structural inconsistencies.

A central claim of the method is that semantic understanding should be explicit rather than implicit. SGDFuse therefore treats SAM-generated masks as direct scene priors and uses a diffusion process for semantic-guided refinement. This design suggests that semantic localization and generative restoration are jointly necessary when the goal is not merely visual sharpness but semantically coherent fused imagery that remains effective for downstream vision tasks (Zhang et al., 7 Aug 2025).

2. Two-stage framework and data flow

The architecture is organized as a two-stage process. Stage I performs preliminary multi-modal feature fusion, extracting modality-specific features from infrared and visible images and producing an initial fused RGB image $F_1 \in \mathbb{R}^{H\times W\times 3}$. Stage II then performs SAM-guided conditional diffusion with coarse-to-fine denoising, using semantic masks from both modalities together with $F_1$ to refine boundaries, detail, and semantic consistency (Zhang et al., 7 Aug 2025).

Stage	Main components	Output
Stage I	MSFEM branch for infrared, Transformer block for visible, cross-attention fusion	Preliminary fused image $F_1$
Stage II	SAM masks, conditional diffusion model, U-Net denoiser, HFAH	Final fused image $I_f$

The key conditioning tensor for Stage II is formed by concatenating the preliminary fused image with two SAM masks: $$I_0 = Concat(F_1, M_{ir}, M_{vis}) \in \mathbb{R}^{H\times W\times 5}.$$ Here $M_{ir} \in \mathbb{R}^{H\times W\times 1}$ and $M_{vis} \in \mathbb{R}^{H\times W\times 1}$ are the SAM-generated masks for infrared and visible images, respectively. The paper attributes three semantic advantages to this construction: target preservation, boundary fidelity, and semantic awareness. In ablation, removing SAM significantly degrades all metrics, which the authors treat as evidence that these masks function as central semantic priors rather than as auxiliary inputs (Zhang et al., 7 Aug 2025).

3. Stage I: preliminary multi-modal feature fusion

Stage I comprises two modality-specific branches. The infrared branch uses a Multi-Scale Feature Enhancement Module (MSFEM), while the visible branch uses a Transformer block with multi-head self-attention to capture global context, fine-grained texture, and long-range dependencies (Zhang et al., 7 Aug 2025).

For the infrared stream, the feature map $F_{ir}$ is processed with four convolutional branches: $$F_i = Conv_{j\times j}(F_{ir}), \quad i=1,2,3,4;\; j=1,3,5,7.$$ This creates multi-scale features with receptive fields spanning shallow detail and broader structural context. The larger-scale branches are concatenated,

$F_{ms} = Concat(F_2, F_3, F_4),$

then enhanced by depthwise and pointwise convolutions,

$F_1$0

The shallow branch is preserved separately and concatenated with the enhanced feature: $F_1$1 A $F_1$2 convolution followed by a sigmoid produces normalized weights, and a residual connection yields the final output: $F_1$3 The paper interprets this as robust multi-scale structure modeling that preserves original infrared information while enhancing salient thermal regions. It further reports that stacking MSFEM three times gives the best tradeoff (Zhang et al., 7 Aug 2025).

The visible branch uses a Transformer block rather than an explicitly convolutional design. The text does not give a formal equation for this block, but states that it is used to extract global context, fine-grained texture, and long-range dependencies from the visible image. Cross-modal fusion is then performed by dynamically aligning and selectively fusing the extracted infrared and visible features through cross-attention interaction, generating the preliminary fused image $F_1$4. This initial output is designed to preserve salient infrared targets and visible detail, but it is not the final reconstruction; its principal role is to provide a structured initialization for the semantic refinement stage (Zhang et al., 7 Aug 2025).

4. Stage II: SAM-guided conditional diffusion and semantic refinement

Stage II is the method’s distinctive contribution. It uses a DDPM-style conditional diffusion process to refine the preliminary fusion under explicit semantic guidance from SAM masks (Zhang et al., 7 Aug 2025). The forward diffusion process gradually adds Gaussian noise to the conditional input $F_1$5: $F_1$6 with closed-form sampling

$F_1$7

where

$F_1$8

The reverse process reconstructs the signal by predicting noise and denoising iteratively: $F_1$9 with

$F_1$0

and mean

$F_1$1

The paper describes this as coarse-to-fine denoising: early steps remove coarse noise and shape global structure, while later steps refine boundaries and local details.

Training uses the simplified DDPM noise-prediction objective

$F_1$2

where $F_1$3. The denoising network $F_1$4 is a U-Net inspired by SR3, with a contracting path of 5 convolutional layers, an expanding path of 5 corresponding convolutional layers, and a diffusion head that predicts noise with the same dimensionality as the input. Because the input has five channels, the denoiser jointly models fused visual appearance, infrared semantics, and visible semantics. The decoder is augmented with a Hierarchical Feature Aggregation Head (HFAH), which aggregates multi-level decoded features, uses spatial attention, strengthens edge, structure, and semantic region modeling, and improves boundary fidelity and detail recovery. The final fused output is

$F_1$5

and the final fusion head uses multiple $F_1$6 convolutions followed by a Tanh activation (Zhang et al., 7 Aug 2025).

5. Losses, supervision, and evaluation protocol

SGDFuse uses separate losses for the two stages. For Stage I, the paper defines a gradient loss

$F_1$7

and an intensity loss

$F_1$8

The manuscript writes the stage-I total loss as

$F_1$9

The accompanying explanation states that this appears to be a typo; based on the preceding definitions, the intended combination is the infrared intensity consistency term and the visible gradient consistency term (Zhang et al., 7 Aug 2025).

Stage II uses mask-guided regional supervision. A joint semantic mask is constructed as

$I_f$0

followed by a mask-guided intensity loss

$I_f$1

and a mask-guided gradient loss

$I_f$2

The stage-II total loss is

$I_f$3

with best weights $I_f$4 and $I_f$5. These losses are region-weighted rather than uniform across pixels, which is consistent with the framework’s emphasis on semantically important regions (Zhang et al., 7 Aug 2025).

Training is conducted in PyTorch on an NVIDIA RTX 4090 using the MSRS dataset, with 1,083 visible-infrared training pairs and 361 test pairs, patch size $I_f$6, Adam optimization, learning rate $I_f$7, batch size 24, and 200 epochs. The model is trained in a two-stage manner: Stage I learns the preliminary fusion, and Stage II learns diffusion-based semantic refinement. At inference, images are processed at original resolution and the model outputs a 3-channel fused image (Zhang et al., 7 Aug 2025).

Evaluation is reported on four public IVIF datasets: MSRS, M$I_f$8FD, LLVIP, and RoadScene. The paper uses seven image-fusion metrics: EN, SD, SF, MI, SCD, VIF, and $I_f$9, with higher values reported as better in the tables. Downstream-task adaptability is evaluated with YOLOv5 for object detection and DeepLabV3+ for semantic segmentation (Zhang et al., 7 Aug 2025).

6. Empirical behavior, ablations, limitations, and nomenclature

The reported experimental results show consistent top-ranked or near-top-ranked performance across datasets. On MSRS, SGDFuse achieves EN $$I_0 = Concat(F_1, M_{ir}, M_{vis}) \in \mathbb{R}^{H\times W\times 5}.$$0, SD $$I_0 = Concat(F_1, M_{ir}, M_{vis}) \in \mathbb{R}^{H\times W\times 5}.$$1, SF $$I_0 = Concat(F_1, M_{ir}, M_{vis}) \in \mathbb{R}^{H\times W\times 5}.$$2, MI $$I_0 = Concat(F_1, M_{ir}, M_{vis}) \in \mathbb{R}^{H\times W\times 5}.$$3, SCD $$I_0 = Concat(F_1, M_{ir}, M_{vis}) \in \mathbb{R}^{H\times W\times 5}.$$4, VIF $$I_0 = Concat(F_1, M_{ir}, M_{vis}) \in \mathbb{R}^{H\times W\times 5}.$$5, and $$I_0 = Concat(F_1, M_{ir}, M_{vis}) \in \mathbb{R}^{H\times W\times 5}.$$6 $$I_0 = Concat(F_1, M_{ir}, M_{vis}) \in \mathbb{R}^{H\times W\times 5}.$$7, with best performance on EN, SF, SCD, VIF, and $$I_0 = Concat(F_1, M_{ir}, M_{vis}) \in \mathbb{R}^{H\times W\times 5}.$$8. On M$$I_0 = Concat(F_1, M_{ir}, M_{vis}) \in \mathbb{R}^{H\times W\times 5}.$$9FD, it is reported as top-ranked on all listed metrics, including EN $M_{ir} \in \mathbb{R}^{H\times W\times 1}$0, SF $M_{ir} \in \mathbb{R}^{H\times W\times 1}$1, MI $M_{ir} \in \mathbb{R}^{H\times W\times 1}$2, SCD $M_{ir} \in \mathbb{R}^{H\times W\times 1}$3, VIF $M_{ir} \in \mathbb{R}^{H\times W\times 1}$4, and $M_{ir} \in \mathbb{R}^{H\times W\times 1}$5 $M_{ir} \in \mathbb{R}^{H\times W\times 1}$6. On LLVIP, it achieves EN $M_{ir} \in \mathbb{R}^{H\times W\times 1}$7, SD $M_{ir} \in \mathbb{R}^{H\times W\times 1}$8, SF $M_{ir} \in \mathbb{R}^{H\times W\times 1}$9, MI $M_{vis} \in \mathbb{R}^{H\times W\times 1}$0, SCD $M_{vis} \in \mathbb{R}^{H\times W\times 1}$1, VIF $M_{vis} \in \mathbb{R}^{H\times W\times 1}$2, and $M_{vis} \in \mathbb{R}^{H\times W\times 1}$3 $M_{vis} \in \mathbb{R}^{H\times W\times 1}$4, with best scores on almost all metrics. On RoadScene, it performs best on MI $M_{vis} \in \mathbb{R}^{H\times W\times 1}$5, SCD $M_{vis} \in \mathbb{R}^{H\times W\times 1}$6, VIF $M_{vis} \in \mathbb{R}^{H\times W\times 1}$7, and $M_{vis} \in \mathbb{R}^{H\times W\times 1}$8 $M_{vis} \in \mathbb{R}^{H\times W\times 1}$9 (Zhang et al., 7 Aug 2025).

The ablation studies attribute the largest degradation to removing SAM. Additional variants remove only the infrared or visible SAM mask, and the results are described as showing that dual-modality semantic masks are complementary. Removing Stage I yields insufficient structural priors and blur or distortion; removing Stage II yields weaker semantic consistency and fine detail; removing diffusion lowers all metrics and degrades clarity, structural integrity, artifact reduction, and high-fidelity reconstruction; removing HFAH worsens boundary fidelity and texture detail. The number of stacked MSFEM and Transformer blocks was varied from 2 to 4, and performance improved up to 3 layers before saturating or slightly declining, leading to a final design with three MSFEM and three Transformer blocks (Zhang et al., 7 Aug 2025).

The downstream-task results are presented as evidence that the fused images are semantically useful rather than only visually plausible. With YOLOv5 on MSRS, SGDFuse improves detection performance over competing fused-image methods, especially for Person and Car. With DeepLabV3+, it reports the best semantic-segmentation mIoU of $F_{ir}$0 across Background, Car, Person, Curve, Color Cone, and Bike. This suggests that semantic-aware fusion can benefit recognition pipelines more directly than methods optimized only for appearance (Zhang et al., 7 Aug 2025).

A stated future direction is real-time optimization, indicating that faster inference remains an open engineering objective. A plausible implication is that performance may depend on mask reliability, because the framework conditions its refinement stage on SAM-generated semantic masks. The method should also be distinguished from similarly named fusion approaches in other domains: Sparse Dense Fusion (SDF) addresses camera-LiDAR fusion for 3D object detection (Gao et al., 2023), and GS-Fuse addresses event-driven financial forecasting through Granger-supervised gated fusion and multi-granularity alignment (Zhang et al., 27 May 2026). In the present usage, SGDFuse specifically denotes a SAM-guided diffusion framework for high-fidelity infrared-visible image fusion (Zhang et al., 7 Aug 2025).