RGB-LWIR Fusion Schema

• RGB-LWIR Fusion Schema is a framework that fuses visible (RGB) and thermal (LWIR) data at pixel, feature, and decision levels to overcome adverse imaging conditions.
• It employs techniques such as alpha blending, attention-based weighting, and confidence mapping to integrate complementary texture, color, and emissivity cues.
• Empirical results indicate that fusion can boost performance metrics by up to 16 percentage points, enhancing applications like object detection, segmentation, and 3D mapping.

RGB-LWIR Fusion Schema encompasses the mathematical frameworks, network architectures, and practical protocols for combining information from red-green-blue (RGB—visible spectrum) and long-wave infrared (LWIR—thermal) sensors across diverse computer vision tasks. Fusing these complementary modalities addresses adverse conditions (e.g., variable illumination, occlusion, atmospheric degradation) by leveraging RGB’s texture and color cues with LWIR’s emissivity-based target localization. The fusion schema includes image-level, feature-level, and decision-level strategies; each is tightly specified in the literature via explicit pipeline diagrams, loss functions, and quantitative performance improvements, covering applications such as object detection, semantic segmentation, salient object detection, 3D mapping, pose estimation, and autonomous navigation.

1. Fusion Schema Fundamentals and Method Taxonomy

Fusion strategies for RGB-LWIR (sometimes referred to as RGBT) are typically categorized into three principal classes:

• Pixel-level/early fusion: Combines registered RGB and LWIR frames via arithmetic operations (e.g., alpha blending, vector scaling) to form a single composite image that is subsequently processed by a convolutional network. For example, simple alpha blending,

$$I_{\rm fused}(x,y) = \alpha\,I_{\rm RGB}(x,y) + (1-\alpha)\,C(L(x,y)),$$

where $C(L(x,y))$ maps the LWIR scalar to a color vector, underpins early-fusion detectors in (Gallagher et al., 2022, Gallagher et al., 23 Dec 2025), and (Sachdeva et al., 24 Dec 2025).

• Feature-level/mid fusion: Processes each modality through a separate encoder, fusing the resulting features via concatenation, weighted summation, or learned attention at specified network depths. Modality-specific confidence weighting, correlation maps, edge-aware guidance, and entropy-based block attention are incorporated to address spatial misalignment and context-adaptiveness (Frigo et al., 2022, Zhou et al., 2021, Vadidar et al., 2022, Liang et al., 2023, Li et al., 2023).
• Decision-level/late fusion: Operates on parallel task heads (e.g., bounding box or heatmap predictions) with dynamic selection or averaging driven by illumination-quality metrics or explicit scalar confidences (Yang et al., 2022, Tang et al., 2022). Late fusion provides robustness under distributional shift or missing modalities.

Hybrid and adaptive strategies further refine this taxonomy by integrating elements such as multi-level fusion, synthetic third modalities (e.g., synthetic SWIR from LWIR), and context-sensitive ratio adaptation.

2. Detailed Fusion Algorithms and Mathematical Formulations

Schematics provided in the literature prescribe the following canonical implementations:

Paper	Fusion Placement	Fusion Operation
(Gallagher et al., 23 Dec 2025)	Early (input)	$$I_\mathrm{fused} = (1-\alpha)I_{RGB} + \alpha\,M(I_{LWIR})$$
(Li et al., 2024)	Early (input, fast)	$\mathbf{c}_f(x) = k(x)\mathbf{c}_v(x)$, $k(x)$ scaling derived from gamma-corrected $I_{LWIR}$
(Frigo et al., 2022)	Mid (feature)	Sequential confidence- and correlation-weighted reweighting: $f_m' = C_m \odot f_m$, then $f''_{ct} = M_{ct} \odot [f_c'\|f_t']$
(Li et al., 2023)	Mid (feature)	Asymmetric encoders + Residual Spatial Fusion: $\hat Z_r = Z_r + W_s^{\rm spat}\odot Z_t$ (with confidence gating)
(Vadidar et al., 2022)	Mid (feature, detection)	Channel-wise concatenation + $1\times1$ conv; feature attention via entropy (EBAM): $A_c(F)\otimes F$, $A_s(F)\otimes F$
(Lysyi et al., 6 Dec 2025)	Late (embedding fusion)	Palette-invariant thermal embedding $\bar z$ fused with contrast-normalized RGB $r$ using a learnable gate $g$: $u = g\odot \bar z + (1-g)\odot r$

A selection of comprehensive examples:

• DooDLeNet (Frigo et al., 2022) merges RGB and LWIR via two ResNet-101 DeepLabV3+ towers, fusing at encoder layers 2/4. Confidence maps $C_m$, built from decoder softmax outputs, reweight each stream before concatenation. A correlation map $M_{ct}$ derived from normalized dot products of decoder logits further reweights the concatenated map, compensating for misalignment. Shared decoder supervision employs multi-branch cross-entropy.
• FCDFusion (Li et al., 2024) operates at the pixel level, scaling the RGB vector at each location by a function $k(x)$ of gamma-corrected, normalized LWIR. This achieves near-identity color preservation with minimal (7 FLOPs/pixel) computation, evaluated by the color deviation metric $\mathrm{CD}(x)$—the angle between original and fused RGB vectors.

3. Sensor Calibration, Alignment, and Preprocessing

All fusion methods presuppose accurate alignment between RGB and LWIR frames at pixel or feature level. Approaches include:

• Rigid co-mounting and identical FoV: Used in aerial scenarios (Gallagher et al., 2022, Gallagher et al., 23 Dec 2025), where frames are aligned by cropping/rescaling. Intrinsic and extrinsic calibration (checkerboard patterns, homography estimation) further refine this co-registration (Sachdeva et al., 24 Dec 2025, Chen et al., 2020, Sharma et al., 2024, Upadhyay et al., 2024).
• Targetless calibration with auxiliary sensors: IRisPath (Sharma et al., 2024) leverages 3D LiDAR intensity images and feature matching (SuperGlue/ORB) to recover the RGB↔IR transformation, achieving sub-centimeter, sub-degree accuracy.
• Edge-based cross-correlation: (Vadidar et al., 2022) achieves alignment by maximizing correlation between Canny edge maps of RGB and rescaled LWIR, then transfers labels via computed warps.

Preprocessing typically includes intensity normalization, color-jitter augmentation of RGB, min–max scaling of LWIR, and, where applicable, colorization of LWIR for display or early fusion.

4. Task-Specific Architectures and Applications

Fusion schema implementation is highly task-dependent:

• Object detection: Early fusion via pixel blending (YOLOv7 (Gallagher et al., 2022), YOLOv8/10/11 (Gallagher et al., 23 Dec 2025)), mid-level feature concatenation with attention (Scaled-YOLOv4+EBAM (Vadidar et al., 2022)), and illumination- or confidence-aware late fusion (YOLOv4+IAN (Yang et al., 2022)). Reported mAP gains for fused over single-modality models range from 2–15 pp, with integration of palette-invariant losses further closing the unimodal–multimodal gap (Lysyi et al., 6 Dec 2025).
• Semantic segmentation: Deep multi-stream encoders with cross-modal confidence/correlation weighting (DooDLeNet (Frigo et al., 2022)), edge-guidance (EGFNet (Zhou et al., 2021)), residual spatial fusion with confidence gates (RSFNet (Li et al., 2023)), and explicit attention-enhanced fusion (EAEF (Liang et al., 2023)) define SOTA segmentation pipelines.
• Salient object detection, crowd counting, pose estimation: EAEF blocks generalize to these tasks (Liang et al., 2023), while LWIRPOSE (Upadhyay et al., 2024) lays out pre-processing and baseline fusion protocols in absence of dedicated fusion networks.
• 3D mapping: (Chen et al., 2020) demonstrates a geometric pipeline wherein dense RGB-based reconstruction is post-processed with thermal frame projection and averaging, yielding thermo-RGB point clouds for robotics.
• Autonomous navigation: IRisPath (Sharma et al., 2024) fuses per-patch ResNet-18 embeddings from RGB and LWIR (plus vehicle speed) into a traversability costmap via MLP regression, with calibration from LiDAR-derived spatial correspondence.

5. Adaptive and Context-Sensitive Fusion

Optimal fusion ratios or fusion strategies are not static:

• Dynamic ratio selection: (Sachdeva et al., 24 Dec 2025) employs a lux sensor to switch between stored YOLO models trained at different RGB/LWIR blend ratios; (Gallagher et al., 23 Dec 2025) establishes optimal $\alpha$ per altitude (h), empirically: $$\alpha_{\mathrm{opt}}(h\leq10\,\mathrm{m})\approx0.2$$–$0.3$, $h\geq15\,\mathrm{m}\to0.1$.
• Gated or softmax attention: (Hussain et al., 15 Oct 2025, Lysyi et al., 6 Dec 2025), and (Liang et al., 2023) parameterize fusion weights via learned gating mechanisms, softmax over global pooled features, or explicit MLP-mapped attention.
• Palette-invariant learning: To prevent colorization bias in thermal data (especially with pseudo-color palettes), (Lysyi et al., 6 Dec 2025) enforces embedding consistency across multiple palette renders of the same thermal frame, using a mean-squared difference penalty.

Ablation studies consistently show enhanced adaptation and robustness—both in mAP and day/night, seasonal, or weather transfer—when fusion parameters are learned/adapted in context.

6. Quantitative Impact and Benchmarks

Empirical results confirm substantial benefits of RGB-LWIR fusion. Table entries below sample reported metrics under various fusion schemas:

Task/Benchmark	Baseline mAP/mIoU	Fused mAP/mIoU	Delta	Reference
Semantic segmentation (MF)	~50.1–50.7%	57.3% (*)	+6.6 pp	(Frigo et al., 2022)
Object detection (PVF-10)	0.78 / 0.74	0.903 (**)	+12–16 pp	(Lysyi et al., 6 Dec 2025)
Landmine detection (Y11)	95.0% (RGB)	97.6% (fusion)	+2.6 pp	(Gallagher et al., 23 Dec 2025)
RGBT tracking (EAO, VOT)	0.3433 (SiamRPN)	0.3986	+5.5 pp	(Tang et al., 2022)
Scaled-YOLOv4 (FLIR, 640)	63.0 / 56.6%	82.9% (fusion)	+10–20 pp	(Vadidar et al., 2022)

() DooDLeNet, full fusion: confidence+correlation weighting. (*) Palette-invariant fusion + adaptive re-acquisition.

Additional observations:

• Aggregated multi-temporal training outperforms season-specific splits by up to +9.6% mAP (Gallagher et al., 23 Dec 2025).
• Under adverse or night conditions, fusion consistently outperforms either RGB or LWIR alone; sometimes only the fused model is effective in both day and night (Gallagher et al., 2022).
• On complexity–throughput tradeoff, practical fusion pipelines such as YOLOv11 train 17.7× faster than transformer-based baselines, while still gaining the majority of accuracy benefit (Gallagher et al., 23 Dec 2025).

7. Limitations and Challenges

Explicit challenges include:

• Calibration/Alignment: Inaccurate registration can degrade fusion benefits; advanced calibration (e.g., (Sharma et al., 2024)) or attention-based misalignment compensation (Frigo et al., 2022) are required for robust performance.
• Domain gap and dataset bias: Sensor-specific artifacts, pseudo-color palette bias in thermal images, or temporal annotations (manual, imperfect) can reduce generalization (Lysyi et al., 6 Dec 2025, Vadidar et al., 2022). Palette-invariance objectives and synthetic augmentation pipelines (CycleGAN (Yang et al., 2022)) partially mitigate these issues.
• Extensibility: While surface-laid landmines exhibit maximal thermal contrast, buried ordnance presents a sharply reduced signal; fusion schema for subsurface detection must incorporate heat-transfer modeling and potentially fuse additional sensor types (Gallagher et al., 23 Dec 2025).
• Computational efficiency: Some approaches (FCDFusion (Li et al., 2024)) favor pixel-level operations to reach 7 FLOPs/pixel, while others (multi-stream DenseNet, SPP, or EBAM) may require more significant compute—real-time applications must balance accuracy with latency (Gallagher et al., 23 Dec 2025, Vadidar et al., 2022).

A plausible implication is that highly adaptive, modular fusion architectures—combining robust cross-modal registration, content-aware weighting, and low-level computation—are likely to define future state-of-the-art under challenging, multimodal vision scenarios.