Visibility-Aware Semantic Fusion

• Visibility-aware semantic fusion is a multimodal integration strategy that adjusts sensor contributions based on local observation reliability and task-critical semantic cues.
• It leverages adaptive weighting mechanisms—involving attention, entropy, and depth cues—to balance inputs from modalities like infrared and visible imagery across diverse applications.
• Empirical studies demonstrate that this approach improves semantic consistency, target detection, and robustness under adverse lighting and environmental conditions.

Visibility-aware semantic fusion denotes a class of multimodal fusion strategies in which the contribution of each modality, view, or sensor is adjusted according to local observation reliability while the fused representation is simultaneously biased toward task-critical semantics. In infrared–visible image fusion, this typically means that infrared should dominate in low-visibility or target-rich regions while visible imagery should dominate where structure and texture are reliable (Sun et al., 14 Sep 2025). In multi-view 3D generation, it appears as confidence-aware fusion in which each viewpoint contributes according to its local observation reliability through attention-entropy weighting and visibility weighting (Li et al., 12 Mar 2026). In autonomous-driving sensor fusion, it is realized by conditioning attentive cross-modal fusion on spatially varying local depth tokens and a global condition token because sensor reliability largely depends on depth (Broedermannn et al., 11 Sep 2025). In RGB-D 3D Gaussian SLAM, it takes the form of geometry-validated semantic evidence fusion, where semantic observations are weighted by visibility consistency, surface-supported boundary evidence, and ray-level conflict uncertainty rather than treated as uniformly valid supervision (Jha et al., 28 Jun 2026).

1. Definition, scope, and conceptual boundaries

Visibility-aware semantic fusion addresses settings in which multimodal observations are complementary but not uniformly trustworthy. In infrared–visible fusion, the infrared modality is robust to low light and adverse weather and gives strong responses on warm objects, while the visible modality offers rich texture, edges, and structure under good lighting conditions; the two can therefore be complementary or even conflicting across regions rather than simply additive (Sun et al., 14 Sep 2025). In outdoor multimodal perception, environmental conditions cause strong local, depth-dependent variations in the signal-to-noise ratio of each sensor, so uniform fusion over the full scene is inadequate (Broedermannn et al., 11 Sep 2025). In semantic 3D Gaussian SLAM, occlusions, unsupported semantic boundaries, and ambiguous ray geometry make 2D semantic priors unequally reliable during online mapping (Jha et al., 28 Jun 2026).

Within this literature, “visibility-aware” does not denote a single mechanism. It can refer to modality selection by attention and blending in image fusion, to explicit geometric visibility checks in 3D latent fusion, to clarity sensing in diffusion models, to local depth-conditioned sensor weighting, or to depth-consistency masks in SLAM. “Semantic” likewise ranges from weak ROI supervision and segmentation-derived features to textual descriptions, detection features, semantic logits, and class-conditioned edge constraints. What unifies these formulations is the replacement of uniform fusion with reliability-aware, task-aware fusion.

A common misconception is that visibility-aware fusion is only a low-light enhancement problem. The literature shows a broader scope: visibility may refer to night scenes and glare in infrared–visible fusion (Pei et al., 2024), geometric visibility of 3D tokens across viewpoints (Li et al., 12 Mar 2026), clarity under blur and haze in conditional diffusion (Jie et al., 11 Sep 2025), spatially varying sensor reliability governed by depth (Broedermannn et al., 11 Sep 2025), or semantic observability under occlusion in SLAM (Jha et al., 28 Jun 2026). This suggests that visibility-aware semantic fusion is best understood as a general design principle for reliability-conditioned semantic integration rather than as a narrow subfield of nighttime imaging.

2. Reliability modeling and visibility-aware weighting

A central mechanism is adaptive weighting. In FusionNet, infrared and visible features are encoded separately as $F_{ir}$ and $F_{vis}$, concatenated, and passed through a modality attention module producing an attention mask $A \in [0,1]^{C \times H \times W}$. The fused feature is

$F_{attn} = A \cdot F_{ir} + (1 - A) \cdot F_{vis},$

and the final image is produced by a pixel-wise alpha map

$$I_{fused}(x,y) = \alpha(x,y)\, I_{ir}(x,y) + (1 - \alpha(x,y))\, I_{vis}^Y(x,y),$$

with brighter alpha-map regions indicating higher infrared contribution (Sun et al., 14 Sep 2025). This yields both spatially adaptive and channel-adaptive visibility selection.

In MV-SAM3D, visibility-aware fusion is performed in 3D latent space rather than image space. For each 3D token $l$, per-view velocity predictions are fused as

$$\hat{v}(x_t, t)[l] = \sum_{i=1}^{N} w_i(l)\, v_\theta(x_t, t, c_i)[l],$$

with weights

$$w_i(l) = (1-\gamma)\, w_i^{\text{ent}(l)} + \gamma\, w_i^{\text{vis}(l)}.$$

Here $w_i^{\text{ent}(l)}$ is derived from cross-attention entropy and $w_i^{\text{vis}(l)}$ from explicit 3D visibility checks using DDA ray tracing over sparse voxels (Li et al., 12 Mar 2026). The same principle reappears in VCS-SLAM, where semantic evidence at pixel $F_{vis}$0 is weighted by

$F_{vis}$1

with

$F_{vis}$2

Visibility consistency suppresses updates under depth mismatch, while conflict-aware uncertainty reduces supervision on ambiguous rays (Jha et al., 28 Jun 2026).

In DGFusion, reliability is conditioned on depth through local tokens rather than explicit per-pixel masks. At each feature level and window, a local depth token is computed as

$F_{vis}$3

and queries for windowed cross-attention concatenate RGB tokens with the local depth token and a global condition token (Broedermannn et al., 11 Sep 2025). In FS-Diff, clarity sensing performs a related role: CA-CLIP and CLSE classify each modality as clear or blurred and select the semantic embedding $F_{vis}$4 from the clearer modality, or an element-wise maximum if both are blurred (Jie et al., 11 Sep 2025). Across these systems, visibility awareness is implemented as a learned or computed trust assignment.

3. Semantic guidance and task alignment

Semantic guidance enters fusion through several distinct channels. FusionNet uses weak ROI supervision: with $F_{vis}$5 denoting pixels inside ground-truth bounding boxes, the target-aware loss is

$F_{vis}$6

Because ROIs correspond to important objects such as pedestrians and vehicles, this pushes the fused result toward infrared in task-critical regions without requiring dense labels or an explicit detector (Sun et al., 14 Sep 2025).

MAFS frames infrared–visible fusion and semantic segmentation as reciprocal promotion. Deep semantic features from the segmentation stream are injected into the fusion stream through Progressive Heterogeneous Fusion, while a pre-trained SegNeXt teacher operating on the fused image supplies knowledge-distillation logits to the segmentation branch. Joint training uses $F_{vis}$7-fairness with $F_{vis}$8 and Dynamic Weight Average to balance fusion and segmentation losses (Wang et al., 15 Sep 2025). The fully semantic-driven method of “Breaking Free from Fusion Rule” goes further by making semantic segmentation the primary learning signal after warm start, using

$F_{vis}$9

and explicitly avoiding hand-crafted fusion losses during semantic training (Wu et al., 2022).

Text-guided variants use language as semantic control. TeSG derives a caption $A \in [0,1]^{C \times H \times W}$0 with BLIP, removes a keyword $A \in [0,1]^{C \times H \times W}$1 to form $A \in [0,1]^{C \times H \times W}$2, and computes mask semantics from diffusion-noise differences,

$A \in [0,1]^{C \times H \times W}$3

then combines mask-guided cross-attention and text-driven attentional fusion to weight infrared and visible features according to both spatial semantics and caption semantics (Zhu et al., 20 Jun 2025). In AMFusion, semantic guidance is detection-driven rather than text-driven: a pre-trained YOLOv5s backbone provides detection features used by the Detection-guided Semantic Fusion Module, and semantic features later modulate spatial reconstruction through the Semantic-Guided Rectify Module (Pei et al., 2024). The dual-modal prior semantic guided method for intelligent transportation systems instead learns significant prior semantics separately for infrared and visible inputs through two parallel segmentation branches with RFaM, then combines low-frequency semantic priors with high-frequency details through MRaF (Li et al., 2024).

These designs differ operationally, but they share a common objective: semantic information is not an auxiliary by-product of fusion; it determines which regions, structures, and boundaries should be preserved, emphasized, or downweighted.

4. Representative formulations across application domains

The diversity of the field is clearer when representative systems are compared side by side.

Domain	Representative formulation	Example
Infrared–visible image fusion	Modality-aware attention, pixel-wise alpha blending, ROI supervision	FusionNet (Sun et al., 14 Sep 2025)
Joint fusion and segmentation	PHF, MST decoder, KD, $A \in [0,1]^{C \times H \times W}$4-fair DWA	MAFS (Wang et al., 15 Sep 2025)
Fully semantic-driven IVF	Warm start, semantic loss, correlation regularization	(Wu et al., 2022)
Text-guided IVF	SIG, MGCA, TDAF with mask and text semantics	TeSG (Zhu et al., 20 Jun 2025)
Nighttime visibility enhancement	IDFM, DSFM, illumination loss, SRM	AMFusion (Pei et al., 2024)
ITS-oriented prior semantic fusion	Dual semantic branches, RFaM, MRaF	(Li et al., 2024)
Adversarially robust IVF	Adaptive Average, HDS, AAT	PAIF (Liu et al., 2023)
Joint fusion and super-resolution	CLSE, CA-CLIP semantics, conditional diffusion	FS-Diff (Jie et al., 11 Sep 2025)
Multi-view 3D generation	Attention-entropy weighting and visibility weighting in 3D latent space	MV-SAM3D (Li et al., 12 Mar 2026)
Autonomous-driving sensor fusion / RGB-D SLAM	Depth-guided local tokens or geometry-validated semantic evidence	DGFusion (Broedermannn et al., 11 Sep 2025); VCS-SLAM (Jha et al., 28 Jun 2026)

This comparison shows that the phrase encompasses image-space blending, feature-space fusion, latent-space velocity fusion, token-conditioned cross-attention, and reliability-weighted map optimization. A plausible implication is that the field is converging not on a single architecture, but on a recurring pattern: semantic cues determine relevance, while visibility cues determine trust.

5. Objectives, interpretability, and empirical evidence

A notable property of many visibility-aware semantic fusion systems is interpretability. FusionNet visualizes the alpha map and states that brighter regions indicate higher contribution from the infrared modality, while darker regions indicate higher contribution from the visible modality (Sun et al., 14 Sep 2025). MV-SAM3D exposes per-token weights from entropy and visibility terms, making view contribution explicit at each flow step (Li et al., 12 Mar 2026). TeSG separates mask semantics from text semantics, so foreground localization and semantic gating are structurally distinct (Zhu et al., 20 Jun 2025). VCS-SLAM produces visibility masks, uncertainty weights, and edge-supported penalties that can be inspected independently (Jha et al., 28 Jun 2026).

The empirical record indicates that these mechanisms improve both low-level and downstream metrics. On the M3FD test set, FusionNet reports SSIM $A \in [0,1]^{C \times H \times W}$5, MSE $A \in [0,1]^{C \times H \times W}$6, Entropy $A \in [0,1]^{C \times H \times W}$7, and ROI-SSIM $A \in [0,1]^{C \times H \times W}$8, supporting the claim that target-aware supervision improves semantic preservation in regions containing important objects (Sun et al., 14 Sep 2025). MAFS reports that DWA with $A \in [0,1]^{C \times H \times W}$9 yields the best mIoU $F_{attn} = A \cdot F_{ir} + (1 - A) \cdot F_{vis},$0 on MFNet while maintaining strong visual metrics, and also reports student mIoU values of $F_{attn} = A \cdot F_{ir} + (1 - A) \cdot F_{vis},$1 on PST900 and $F_{attn} = A \cdot F_{ir} + (1 - A) \cdot F_{vis},$2 on FMB (Wang et al., 15 Sep 2025). TeSG reports best detection performance on MSRS with mAP@0.5 $F_{attn} = A \cdot F_{ir} + (1 - A) \cdot F_{vis},$3, [email protected] $F_{attn} = A \cdot F_{ir} + (1 - A) \cdot F_{vis},$4, [email protected]:0.95 $F_{attn} = A \cdot F_{ir} + (1 - A) \cdot F_{vis},$5, and best semantic segmentation mIoU $F_{attn} = A \cdot F_{ir} + (1 - A) \cdot F_{vis},$6, indicating that textual semantic guidance improves target visibility for detection and segmentation (Zhu et al., 20 Jun 2025).

The same pattern holds outside infrared–visible fusion. DGFusion reports PQ $F_{attn} = A \cdot F_{ir} + (1 - A) \cdot F_{vis},$7 and mIoU $F_{attn} = A \cdot F_{ir} + (1 - A) \cdot F_{vis},$8 on MUSES, with the largest gains over CAFuser under adverse conditions such as fog, rain, and snow; its ablations show that the auxiliary depth head, local depth tokens, and robust depth loss all contribute (Broedermannn et al., 11 Sep 2025). VCS-SLAM reports mIoU $F_{attn} = A \cdot F_{ir} + (1 - A) \cdot F_{vis},$9 on Replica, Boundary F1 $$I_{fused}(x,y) = \alpha(x,y)\, I_{ir}(x,y) + (1 - \alpha(x,y))\, I_{vis}^Y(x,y),$$0, and Depth-Inconsistent mIoU $$I_{fused}(x,y) = \alpha(x,y)\, I_{ir}(x,y) + (1 - \alpha(x,y))\, I_{vis}^Y(x,y),$$1, directly linking geometry-validated semantic evidence fusion to better semantic consistency and boundary preservation (Jha et al., 28 Jun 2026). In adversarial settings, PAIF reports gains of $$I_{fused}(x,y) = \alpha(x,y)\, I_{ir}(x,y) + (1 - \alpha(x,y))\, I_{vis}^Y(x,y),$$2 mIOU of segmentation in the adversarial scene and frames robust fusion as a perception-aware problem in which architecture and learning strategy must defend downstream perception rather than merely maximize visual quality (Liu et al., 2023).

6. Limitations, tensions, and open directions

The literature also makes clear that visibility-aware semantic fusion is not solved. In MV-SAM3D, entropy-based confidence can fail for symmetric or repetitive textures, and visibility weighting depends on the correctness of Stage 1 geometry; unobserved regions remain generative (Li et al., 12 Mar 2026). In DGFusion, depth is only a proxy for local sensor reliability, lidar is sparse and noisy in adverse conditions, and calibration quality remains critical (Broedermannn et al., 11 Sep 2025). In VCS-SLAM, the current experiments assume relatively reliable 2D labels, and future work is directed toward robustness to noisy semantic predictions from open-vocabulary or pre-trained 2D segmentation models (Jha et al., 28 Jun 2026).

Text-guided and diffusion-based approaches introduce their own dependencies. TeSG relies on BLIP and a frozen diffusion model, so inaccurate captions or noise predictions can corrupt mask semantics; the current formulation hinges on one keyword $$I_{fused}(x,y) = \alpha(x,y)\, I_{ir}(x,y) + (1 - \alpha(x,y))\, I_{vis}^Y(x,y),$$3, which limits complex scenes with many target classes (Zhu et al., 20 Jun 2025). FS-Diff uses global clarity labels rather than spatial clarity maps, depends on synthetic high-resolution fused targets produced offline by CDDFuse, and incurs substantial computational cost, with about $$I_{fused}(x,y) = \alpha(x,y)\, I_{ir}(x,y) + (1 - \alpha(x,y))\, I_{vis}^Y(x,y),$$4M parameters, about $$I_{fused}(x,y) = \alpha(x,y)\, I_{ir}(x,y) + (1 - \alpha(x,y))\, I_{vis}^Y(x,y),$$5k GFLOPs at $$I_{fused}(x,y) = \alpha(x,y)\, I_{ir}(x,y) + (1 - \alpha(x,y))\, I_{vis}^Y(x,y),$$6 and $$I_{fused}(x,y) = \alpha(x,y)\, I_{ir}(x,y) + (1 - \alpha(x,y))\, I_{vis}^Y(x,y),$$7, and inference of about $$I_{fused}(x,y) = \alpha(x,y)\, I_{ir}(x,y) + (1 - \alpha(x,y))\, I_{vis}^Y(x,y),$$8 seconds per image (Jie et al., 11 Sep 2025).

A second tension concerns supervision. Fully semantic-driven fusion and dual-branch prior-semantic methods rely on semantic labels, which limits applicability where dense annotations are scarce (Wu et al., 2022). ITS-oriented prior semantic fusion similarly depends on annotated task structure and is tied to task categories such as road, background, pedestrian, car, and bike (Li et al., 2024). This suggests that future work will likely continue moving toward weaker supervision, more explicit reliability estimation, and broader task transfer.

The broadest lesson across these papers is that semantic fusion alone is insufficient when observation reliability varies sharply across space, views, or time, and visibility reasoning alone is insufficient when the system does not know which regions matter. Visibility-aware semantic fusion therefore emerges as a joint answer to both problems: it asks not only what information is available, but whether it is trustworthy here, now, and for this task.