IRDFusion: Diverse Infrared–Visible Fusion Techniques

• IRDFusion is a term for varied fusion approaches that integrate infrared and visible data, emphasizing thermal saliency alongside structural cues.
• Its infrared-centric methods use asymmetric architectures where IR leads and RGB is distilled, enabling enhanced multispectral detection.
• The concept extends to iterative feedback, 3D mapping, and missing-modality challenges, addressing real-world sensor and registration issues.

IRDFusion denotes several infrared–visible fusion formulations rather than a single universally fixed architecture. In recent arXiv usage it can describe an “infrared-driven / infrared-dominant fusion” interpretation aligned with Infrared-Centric Fusion for multispectral detection (Hwang et al., 21 May 2025), the formally named “Iterative Relation-Map Differential Guided Feature Fusion” framework for multispectral object detection (Shen et al., 11 Sep 2025), a dense 3D thermal–RGB mapping pipeline for firefighting robots (Chen et al., 2020), and a dictionary-guided missing-IR fusion framework also called IRDFusion or DCMIF (Zhang et al., 9 Mar 2026). Across these usages, the recurring problem is how to preserve thermal saliency while retaining the structural, semantic, or geometric information supplied by visible imagery.

1. Terminology and scope

The literature uses the label in multiple, non-equivalent ways. In some papers it names a specific architecture; in others it functions as a descriptive label for infrared-driven fusion behavior. The following summary captures the main usages explicitly documented in the cited works.

Usage	Core mechanism	Representative paper
Infrared-driven / infrared-dominant fusion	IR as primary modality; RGB distilled and gated into IR-centric features	(Hwang et al., 21 May 2025)
Iterative Relation-Map Differential Guided Feature Fusion	MFRM + DFFM with iterative feedback and relation-map differences	(Shen et al., 11 Sep 2025)
Dense 3D thermal–RGB fusion	SfM/MVS RGB reconstruction with IR projection and averaged thermal values	(Chen et al., 2020)
Missing-IR coefficient-domain fusion	Shared dictionary, VIS-guided IR inference, adaptive coefficient fusion	(Zhang et al., 9 Mar 2026)

A recurrent source of confusion is the assumption that IRDFusion names one canonical model. The papers do not support that reading. Instead, the term is reused across multispectral detection, 3D mapping, and missing-modality fusion. Another misconception is that IRDFusion always implies symmetric RGB–IR treatment. In the infrared-centric strand, the stated design principle is explicitly asymmetric: IR carries the main structural and localization responsibility, while RGB is compressed, distilled, and selectively injected (Hwang et al., 21 May 2025).

2. Infrared-centric multispectral detection

In the multispectral object-detection literature, IRDFusion is closely associated with the infrared-centric design philosophy formalized as IC-Fusion. The core claim is derived from wavelet analysis on aligned RGB–IR pairs: RGB information is mostly concentrated in the $LL$ band, while IR $LH/HL$ bands show sharper contours and object boundaries. This motivates an asymmetric detector in which IR is the driver modality and RGB is treated as a complementary semantic channel. In single-modality training on FLIR-aligned with RT-DETR, the reported baselines were RGB-only $\text{mAP} = 33.5$ and IR-only $\text{mAP} = 43.6$, with similar trends for Faster-RCNN and YOLOv5 (Hwang et al., 21 May 2025).

The resulting architecture is a multispectral detection transformer based on RT-DETR with dual backbones, an infrared-centric fusion module, and a DETR-style encoder–decoder. The backbone asymmetry is explicit: RGB uses ResNet-18 and IR uses ResNet-50. Multi-scale features from stages 3–5 are extracted from both streams. RGB features are first refined by Multi-Scale Feature Distillation (MSFD), then fused with IR by a three-stage block composed of Cross-Modal Channel Shuffle Gate (CCSG), Cross-Modal Large Kernel Gate (CLKG), and Channel Shuffle Projection (CSP). The fused features are fed directly to the RT-DETR encoder: $$\mathbf{F}^{(l)}_{\text{fused}} = \mathcal{F}\!\left(\hat{\mathbf{F}}^{(l)}_{\text{rgb}}, \mathbf{F}^{(l)}_{\text{ir}}\right).$$

MSFD is applied only to RGB. It uses $3 \times 3$, $5 \times 5$, and dilated $5 \times 5$ depthwise branches, concatenates them with the original RGB feature, and distills the result with CSP. CCSG then performs channel shuffle, grouped $1\times1$ convolution, GELU gating, and residual addition for lightweight channel-wise cross-modal interaction. CLKG applies standard $5\times5$ and dilated $LH/HL$0 depthwise convolutions to model long-range spatial context, again with GELU gating and a residual path. CSP acts as the compact projection stage in both RGB distillation and cross-modal fusion.

The training strategy is also asymmetric in effect while remaining loss-simple. The detector uses RT-DETR’s standard Hungarian matching with classification and box regression losses, with no additional modality-specific losses. Instead, a two-stage schedule is adopted from GM-DETR: first train RGB-only and IR-only RT-DETR models, then fine-tune the full dual-stream fusion model on paired RGB–IR data. This is presented as a response to limited aligned multispectral data.

Quantitatively, the full IC-Fusion configuration with RGB ResNet18 and IR ResNet50 reported FLIR-aligned results of mAP50 $LH/HL$1, mAP75 $LH/HL$2, and mAP $LH/HL$3, and LLVIP results of mAP50 $LH/HL$4, mAP75 $LH/HL$5, and mAP $LH/HL$6. On FLIR backbone ablation, RGB ResNet18 + IR ResNet50 achieved mAP $LH/HL$7 with 56.04M parameters and 87.97G MACs, while RGB ResNet50 + IR ResNet18 gave mAP $LH/HL$8 at the same complexity. This is the paper’s central empirical argument that assigning more capacity to RGB is not helpful and that IR should be the primary modality (Hwang et al., 21 May 2025).

3. “IRDFusion” as iterative relation-map differential guidance

A separate 2025 line of work uses IRDFusion as the formal model name: “IRDFusion: Iterative Relation-Map Difference guided Feature Fusion for Multispectral Object Detection” (Shen et al., 11 Sep 2025). Its problem statement is different from the infrared-centric strand. Rather than arguing that IR should dominate, it argues that existing multispectral fusion methods preserve modality-shared background clutter and fail to explicitly suppress common-mode noise. The proposed response is an iterative framework centered on two modules: the Mutual Feature Refinement Module (MFRM) and the Differential Feature Feedback Module (DFFM).

MFRM starts from transformer-style self-attention in each modality. For visible and thermal feature sequences $LH/HL$9 and $\text{mAP} = 33.5$0, the relation maps are

$\text{mAP} = 33.5$1

Instead of direct self-attention output, MFRM fuses Value tensors with learnable cross-modal coefficients: $\text{mAP} = 33.5$2 and then applies each modality’s own relation map: $\text{mAP} = 33.5$3 This is presented as preserving intra-modal structure while injecting cross-modal content.

DFFM introduces the paper’s differential-amplifier analogy. A differential feature is computed, for example,

$\text{mAP} = 33.5$4

and fed back into the next iteration: $\text{mAP} = 33.5$5 The paper further rewrites the differential term in terms of relation-map differences,

$\text{mAP} = 33.5$6

making the “relation-map difference guided” description literal rather than metaphorical.

IRDFusion is implemented as a plug-in module for both YOLOv5 and Co-DETR / Double-Co-DETR. It operates on flattened, position-encoded paired features, iterates MFRM and DFFM, and then returns refined features to the detector neck and heads. The reported best iteration count is four: fewer iterations leave redundant features insufficiently suppressed, whereas more than four degrade performance by weakening differential features and amplifying background noise.

The main reported results are on FLIR, LLVIP, and M$\text{mAP} = 33.5$7FD. Against the Double-Co-DETR + NiNfusion baseline, IRDFusion improved FLIR from mAP50 $\text{mAP} = 33.5$8, mAP75 $\text{mAP} = 33.5$9, mAP $\text{mAP} = 43.6$0 to mAP50 $\text{mAP} = 43.6$1, mAP75 $\text{mAP} = 43.6$2, mAP $\text{mAP} = 43.6$3. On LLVIP it improved the baseline from mAP50 $\text{mAP} = 43.6$4, mAP75 $\text{mAP} = 43.6$5, mAP $\text{mAP} = 43.6$6 to $\text{mAP} = 43.6$7, $\text{mAP} = 43.6$8, and $\text{mAP} = 43.6$9. On M$$\mathbf{F}^{(l)}_{\text{fused}} = \mathcal{F}\!\left(\hat{\mathbf{F}}^{(l)}_{\text{rgb}}, \mathbf{F}^{(l)}_{\text{ir}}\right).$$0FD it improved mAP50 from $$\mathbf{F}^{(l)}_{\text{fused}} = \mathcal{F}\!\left(\hat{\mathbf{F}}^{(l)}_{\text{rgb}}, \mathbf{F}^{(l)}_{\text{ir}}\right).$$1 to $$\mathbf{F}^{(l)}_{\text{fused}} = \mathcal{F}\!\left(\hat{\mathbf{F}}^{(l)}_{\text{rgb}}, \mathbf{F}^{(l)}_{\text{ir}}\right).$$2 and mAP from $$\mathbf{F}^{(l)}_{\text{fused}} = \mathcal{F}\!\left(\hat{\mathbf{F}}^{(l)}_{\text{rgb}}, \mathbf{F}^{(l)}_{\text{ir}}\right).$$3 to $$\mathbf{F}^{(l)}_{\text{fused}} = \mathcal{F}\!\left(\hat{\mathbf{F}}^{(l)}_{\text{rgb}}, \mathbf{F}^{(l)}_{\text{ir}}\right).$$4. The trade-off is computational: YOLOv5 with IRDFusion drops from 45.3 FPS to 17.0 FPS and rises from 75.4M to 134.0M parameters; Co-DETR rises from 485.3M to 510.5M parameters with FPS dropping from 3.6 to 3.1 (Shen et al., 11 Sep 2025).

This strand is important because it is not simply a rebranding of infrared-centric fusion. It frames the central issue as explicit suppression of modality-shared background and iterative amplification of complementary cues. That distinction matters when interpreting “IRDFusion” in multispectral detection papers.

4. Geometric and three-dimensional formulations

The oldest usage in the cited set treats IRDFusion as a 3D mapping pipeline rather than a 2D detector. “3D Fusion of Infrared Images with Dense RGB Reconstruction from Multiple Views -- with Application to Fire-fighting Robots” constructs a single dense 3D map in which every 3D point carries RGB appearance and a thermal value (Chen et al., 2020). The pipeline comprises rigidly mounted stereo RGB cameras and a Tamarisk 320 thermal imager, checkerboard-based intrinsic and extrinsic calibration, VisualSFM for structure from motion, PMVS/CMVS for dense reconstruction, metric scale recovery from the stereo baseline, and thermal assignment by projecting each 3D point into all visible IR frames and averaging sampled IR intensities: $$\mathbf{F}^{(l)}_{\text{fused}} = \mathcal{F}\!\left(\hat{\mathbf{F}}^{(l)}_{\text{rgb}}, \mathbf{F}^{(l)}_{\text{ir}}\right).$$5 The final representation is a dense 3D point cloud with coordinates, RGB color, and thermal intensity. The paper emphasizes practical use for victim localization and hot-spot identification in fire-fighting robots rather than detector training.

A later geometric extension moves from fused 3D mapping to fused 3D rendering. “Fusing in 3D: Free-Viewpoint Fusion Rendering with a 3D Infrared-Visible Scene Representation” introduces Infrared-Visible Gaussian Fusion (IVGF), which reconstructs separate visible and IR 3D Gaussian scenes, concatenates them, and learns a Cross-Modal Adjustment (CMA) module that modulates Gaussian opacity to resolve cross-modal conflicts (Yang et al., 19 Jan 2026). The fused rendering equation is

$$\mathbf{F}^{(l)}_{\text{fused}} = \mathcal{F}\!\left(\hat{\mathbf{F}}^{(l)}_{\text{rgb}}, \mathbf{F}^{(l)}_{\text{ir}}\right).$$6

where $$\mathbf{F}^{(l)}_{\text{fused}} = \mathcal{F}\!\left(\hat{\mathbf{F}}^{(l)}_{\text{rgb}}, \mathbf{F}^{(l)}_{\text{ir}}\right).$$7 is predicted by a small MLP from concatenated IR and visible spherical-harmonics coefficients. On RGBT-Scenes, IVGF reported best SSIM in all scenes, best LPIPS in most scenes, memory of about 21 MB, and about 412 FPS, outperforming a render-then-fuse pipeline that required 69 MB and 239 FPS.

Taken together, these geometric formulations broaden IRDFusion from image-space combination to explicit scene representations. In this usage, the fused entity is not merely a 2D image but a 3D point cloud or Gaussian field supporting spatial reasoning, viewpoint changes, and metric localization.

5. Missing-modality and misregistration settings

One recent direction asks what IRDFusion means when infrared input is absent at inference. “Missing No More: Dictionary-Guided Cross-Modal Image Fusion under Missing Infrared” proposes a coefficient-domain framework built on a shared convolutional dictionary and refers to the framework as IRDFusion or DCMIF (Zhang et al., 9 Mar 2026). Its pipeline is organized as Joint Shared-dictionary Representation Learning (JSRL), VIS-Guided IR Inference (VGII), and Adaptive Fusion via Representation Inference (AFRI). JSRL learns a shared dictionary $$\mathbf{F}^{(l)}_{\text{fused}} = \mathcal{F}\!\left(\hat{\mathbf{F}}^{(l)}_{\text{rgb}}, \mathbf{F}^{(l)}_{\text{ir}}\right).$$8 and coefficient maps $$\mathbf{F}^{(l)}_{\text{fused}} = \mathcal{F}\!\left(\hat{\mathbf{F}}^{(l)}_{\text{rgb}}, \mathbf{F}^{(l)}_{\text{ir}}\right).$$9; VGII infers pseudo-IR coefficients from visible coefficients with a Representation Inference Network and a one-step closed-loop refinement guided by a frozen Qwen-7B-Chat model; AFRI fuses visible and inferred IR coefficients through a Reasoning Fusion Network: $3 \times 3$0 On MSRS, FLIR, and KAIST, the method reported top metrics such as $3 \times 3$1, $3 \times 3$2, and $3 \times 3$3 on MSRS, and it improved a VIS-only object-detection baseline on M$3 \times 3$4FD from mAP $3 \times 3$5 to $3 \times 3$6.

A second direction treats imperfect registration as the core obstacle. “Uncertainty-aware Spatial-Frequency Registration and Fusion for Infrared and Visible Images” proposes SFRF, which combines Multi-scale Iterative Registration (MIR), uncertainty estimation, frequency-domain thermal radiation consistency, and Dual-branch Spatial-Frequency Fusion (DSFF) (Li et al., 13 May 2026). MIR estimates multi-scale deformation fields with Monte Carlo dropout to obtain both mean flow and uncertainty maps, refines flows iteratively with a GRU-based Field Refinement Block, and fuses them with uncertainty- and correlation-aware weights: $3 \times 3$7 Registration is additionally constrained by KL divergence between the Fourier amplitude distributions of registered and undistorted IR images. On M$3 \times 3$8FD, SFRF reported NCC $3 \times 3$9, RMSE $5 \times 5$0, and MEE $5 \times 5$1, and on fusion it obtained the best CC across all reported datasets.

A related but distinct registration-and-fusion framework is BusReF, which explicitly emphasizes reconstructible regions. It trains a shared auto-encoder “bus,” predicts affine and dense deformation fields from the same feature set, and uses a reconstructible mask $5 \times 5$2 in both training and evaluation so that losses do not force the model to explain non-reconstructible regions (Zhang et al., 2023). Its masked NCC term is

$5 \times 5$3

and the full registration objective adds a mask-weighted gradient term derived through the fusion network. On RoadScene, BusReF reported MNCC $5 \times 5$4, substantially above the reported values for LoFTR, SemLA, and MURF.

These works show that IRDFusion is no longer limited to the ideal paired-input setting. Missing-IR inference, uncertainty-aware warping, and reconstructibility-aware training extend the concept into operational regimes where sensors fail, fields of view differ, or strict calibration is unavailable.

6. Broader variants, datasets, and application domains

The wider infrared–visible fusion literature surrounding IRDFusion includes architectures that do not always use the name explicitly but define the surrounding design space. DFVO, for example, addresses low-light visible degradation by learning visible–infrared disentanglement and fusion all at once through a Latent-Common Feature Extractor with a Details-Extraction Module and a Hyper Cross-Attention Module. On LLVIP it reported PSNR $5 \times 5$5 and CC $5 \times 5$6, together with YOLOv5 pedestrian-detection gains after fusion (Zhou et al., 7 May 2025). SPDFusion moves the problem from Euclidean feature space to the manifold of symmetric positive definite matrices, using cross-modal covariance blocks and SPD-based attention; on TNO it reported the best EN, SD, SF, VIF, and AG among the compared methods (Kang et al., 2024). In extreme-scene photogrammetry, “Perceptual Region-Driven Infrared-Visible Co-Fusion for Extreme Scene Enhancement” combines a spatially varying exposure camera, region perception, Retinex-based multi-exposure fusion, and SSIM-guided regional compensation, with a micro-attenuator transmittance ratio of $5 \times 5$7 and about 52 dB additional dynamic range (Tao et al., 6 Dec 2025).

Application-specific systems further diversify the term’s operational meaning. A drone detection pipeline uses two YOLOv4 detectors, an Illumination-Aware Network for late decision fusion, and synthetic IR generation from AIRSim and CycleGAN; the deployed system on a Jetson Xavier reportedly required about 28 milliseconds per RGB/IR image pair (Yang et al., 2022). In wildfire imagery, FIRe-GAN extends an unmatched IR–VIS GAN with a U-Net $5 \times 5$8, a 3-channel $5 \times 5$9, spectral normalization, and a wildfire-specific visible-weighting factor $5 \times 5$0, improving artificial IR generation and fire-scene fusion after transfer learning (Ciprián-Sánchez et al., 2021).

A final expansion is from fusion networks to foundation-model resources. FusionRS introduces 600k aligned RGB–IR–text triplets for remote sensing, produced by translating public RGB imagery into infrared-style counterparts and pairing them with conventional and IR-aware captions (Han et al., 15 Jun 2026). Its tri-modal training objective jointly aligns RGB, IR, and text, while the IR-aware caption subset is used to strengthen infrared-language grounding. This suggests that the term IRDFusion increasingly touches not only image formation and detection, but also representation learning and multimodal supervision.

Across these strands, the literature does not converge on a single definition. What it does converge on is a stable technical agenda: extracting thermal saliency from IR, preserving complementary structure from visible imagery, and doing so under increasingly realistic conditions—adverse illumination, missing modalities, misregistration, 3D scene geometry, and downstream detection or language supervision.