Depth-Guided Feature Fusion (DGF2)

• DGF2 is a depth-conditioned fusion mechanism that injects geometric priors through Spatial Feature Transform and consistency-aware gating.
• It improves low-light semantic segmentation by correcting weak activation maps and enhancing boundary recovery using depth as a structural guide.
• The approach extends to multimodal vision tasks, enabling adaptive sensor fusion with minimal computational overhead and robust performance under challenging conditions.

Searching arXiv for the cited DGF2-related papers to ground the article in current sources. arXiv search results used for grounding:

• (Wang et al., 10 Jul 2025) — "Diffusion-Guided Knowledge Distillation for Weakly-Supervised Low-Light Semantic Segmentation"
• (Broedermannn et al., 11 Sep 2025) — "DGFusion: Depth-Guided Sensor Fusion for Robust Semantic Perception"
• (Lin et al., 14 Oct 2025) — "CurriFlow: Curriculum-Guided Depth Fusion with Optical Flow-Based Temporal Alignment for 3D Semantic Scene Completion"
• (Tang et al., 2019) — "Learning Guided Convolutional Network for Depth Completion"
• (Su et al., 2021) — "Deep feature selection-and-fusion for RGB-D semantic segmentation"
• (Tomar et al., 2022) — "Hybrid Transformer Based Feature Fusion for Self-Supervised Monocular Depth Estimation"
• (Wang et al., 2020) — "Data-Level Recombination and Lightweight Fusion Scheme for RGB-D Salient Object Detection"
• (Zhang et al., 11 Mar 2025) — "Language-Depth Navigated Thermal and Visible Image Fusion"
• (Shivakumar et al., 2019) — "DFuseNet: Deep Fusion of RGB and Sparse Depth Information for Image Guided Dense Depth Completion" Depth-Guided Feature Fusion (DGF2) denotes a family of depth-conditioned fusion mechanisms in which depth, depth-derived features, or depth-supervised latent variables are used to modulate feature extraction, alignment, and aggregation. In "Diffusion-Guided Knowledge Distillation for Weakly-Supervised Low-Light Semantic Segmentation" (Wang et al., 10 Jul 2025), DGF2 is a named module that injects depth maps as illumination-invariant geometric priors into a low-light student backbone through Spatial Feature Transform modulation and a consistency-aware gate. In related work, the same label, or explicitly "DGF2-like" interpretations, is used for dynamic guided convolution, residual RGB-D fusion, curriculum-guided volumetric fusion, local depth-token conditioning, and depth-supervised image fusion (Tang et al., 2019, Su et al., 2021, Lin et al., 14 Oct 2025, Broedermannn et al., 11 Sep 2025, Zhang et al., 11 Mar 2025). This suggests that DGF2 is best understood as a recurring architectural pattern rather than a single canonical operator.

1. Terminology, scope, and motivation

The immediate motivation for DGF2 in DGKD-WLSS is low-light weakly supervised semantic segmentation. Low-light images suffer from low contrast, noise, and color shifts, which compromise boundary visibility and object structure; in weakly supervised settings that rely on Class Activation Maps, these degradations yield unreliable, low-confidence activations and ambiguous pseudo-labels (Wang et al., 10 Jul 2025). DGF2 addresses this by injecting depth maps as illumination-invariant geometric priors, on the premise that depth preserves scene geometry regardless of lighting.

A broader motivation appears across multimodal vision. In dense depth completion, prior work often fused LiDAR and RGB by feature concatenation or element-wise addition, which the guided convolution literature identifies as a naïve strategy for heterogeneous modalities (Tang et al., 2019). In robust sensor fusion for semantic perception, reliability varies spatially across the scene and depends strongly on depth and environmental condition, motivating depth-aware rather than spatially uniform fusion (Broedermannn et al., 11 Sep 2025). In RGB-D semantic segmentation, salient object detection, monocular depth estimation, and thermal-visible fusion, the same general problem recurs: depth is valuable not merely as another channel, but as a structural or reliability prior that should shape how other features are fused (Su et al., 2021, Tomar et al., 2022, Wang et al., 2020, Zhang et al., 11 Mar 2025).

The terminology is not uniform. Some papers explicitly define DGF2 as a module inside a larger framework, whereas others state that the paper does not use the term, or describe a mechanism as "DGF2-like" (Su et al., 2021, Tomar et al., 2022). A common misconception is therefore that DGF2 always refers to the same architectural primitive. The literature represented here instead spans SFT modulation, dynamic kernels, residual cross-modality fusion, local depth tokens in cross-attention, data-level recombination, and auxiliary depth supervision.

2. DGF2 in weakly supervised low-light semantic segmentation

In DGKD-WLSS, DGF2 is the structural arm of a two-part framework that combines Diffusion-Guided Knowledge Distillation with Depth-Guided Feature Fusion (Wang et al., 10 Jul 2025). The overall system uses SSSS with a WideResNet-38 backbone for both teacher and student. The teacher is trained on normal-light images and supplies multi-level features and masks for distillation. DGF2 is inserted only into the low-light student’s feature stream; it does not require depth for the teacher and does not directly interact with normal-light inputs.

The depth source is external monocular depth estimation. Depth maps $I_{depth}$ are predicted per low-light RGB image using Depth Anything, with ZoeDepth and MiDaS also evaluated for robustness. The depth maps correspond to the same image as the low-light RGB input and are fed to a depth prior encoder $M(\cdot)$ at the feature resolution of the current network stage, so the resulting depth features are spatially aligned to the student feature map $F_d$ being fused.

Architecturally, DGF2 is integrated into the student backbone as a set of stage-wise fusion blocks, either single or multiple, forming a coarse-to-fine modulation chain. At selected resolution levels, described as intermediate layers analogous to “Layer 3” and “Layer 6” in distillation, DGF2 fuses depth priors with the current low-light feature map $F_d$ at that stage, producing geometry-aware fused features $F_{fuse}$ that are then propagated to deeper stages and the segmentation head. The multi-stage variant yields the best segmentation results.

The module exists to compensate for two specific low-light failure modes. First, unreliable CAMs produce activation spillover and weak localization. Second, semantically ambiguous pseudo-labels degrade structural learning. DGF2 addresses both by supplying depth-based boundary and shape cues before diffusion-guided distillation further aligns low-light features to normal-light teacher features. In the paper’s formulation, semantic alignment is handled by DGKD, whereas structural reinforcement is handled by DGF2 (Wang et al., 10 Jul 2025).

3. Core operators and mathematical formulation

DGF2 in DGKD-WLSS has two sub-steps: a FiLM-like spatial modulation implemented through SFT, and a consistency-aware attention gate that suppresses irrelevant depth cues while preserving regions jointly supported by RGB and depth features (Wang et al., 10 Jul 2025).

Depth prior extraction is defined as

$prior = M(I_{depth}),$

where $M(\cdot)$ denotes a stack of convolution + ReLU layers that produces a spatial prior feature tensor aligned with the student feature map size. From this prior, two small convolutional stacks generate scale and shift parameters:

$$(\beta, \gamma) = (conv(conv(prior)),\ conv(conv(prior))).$$

These parameters modulate the low-light feature map through SFT:

$$F_{\bar d} = SFT(F_d \mid \beta, \gamma) = \beta \odot F_d + \gamma.$$

Here $\odot$ denotes element-wise multiplication. This stage injects geometric structure from depth into $M(\cdot)$0 by spatially scaling and shifting the features.

Because pure depth priors can introduce irrelevant background information, DGF2 adds a consistency-aware gate. The original and modulated features are converted to per-feature activations using sigmoid:

$M(\cdot)$1

The consistency map is then

$M(\cdot)$2

with $M(\cdot)$3 by default and $M(\cdot)$4 denoting element-wise product. The second term highlights regions where both streams agree; the first term moderates influence in areas where both streams are inactive, mitigating spurious activation. Final fusion is

$M(\cdot)$5

This operator differs from SFT-only modulation because gating is not derived from depth alone. It is derived from the agreement structure between the original low-light features and the depth-modulated features. In the paper’s interpretation, this preserves fine structures while suppressing noise or irrelevant depth cues. In the multi-stage variant, the same fusion is applied hierarchically at several backbone layers in a coarse-to-fine manner (Wang et al., 10 Jul 2025).

A related but architecturally distinct realization of DGF2 appears in guided convolution for depth completion. There, the fusion operator predicts content-dependent, spatially-variant convolution kernels from RGB guidance features and applies them to sparse depth features:

$M(\cdot)$6

with a factorized implementation that reduces complexity from $M(\cdot)$7 to $M(\cdot)$8 and, in the reported example, reduces memory from 10.7 GB to 0.08 GB (Tang et al., 2019). The operator is different, but the design principle is analogous: depth-related fusion is made adaptive and content-aware rather than static.

4. Optimization, supervision, and interaction with distillation

DGF2 in DGKD-WLSS has no separate supervision; it is trained end-to-end under the framework’s overall objective (Wang et al., 10 Jul 2025). The complete loss is

$M(\cdot)$9

with $F_d$0 in experiments. Here $F_d$1 is the image-level classification loss, $F_d$2 is the self-supervised segmentation loss, $F_d$3 is the diffusion training loss at scale $F_d$4, and $F_d$5 is the distillation loss aligning denoised student features and masks to teacher normal-light counterparts.

The weak supervision pipeline uses image-level labels only. CAMs are produced by the student’s classification head, and PAMR is applied to refine CAMs into pseudo-masks which then supervise segmentation through $F_d$6. Since the student backbone contains DGF2, the fused features $F_d$7 affect both the segmentation head and the multi-level distillation targets. DGF2 therefore benefits simultaneously from improved pseudo-mask supervision and from normal-light semantic alignment.

DGKD treats low-light features as noisy variants of normal-light features. A diffusion model is trained on normal-light teacher features and masks and is used to denoise student features; the reported diffusion schedule uses one forward noise pass on teacher features and masks and five denoising passes on the student, with $F_d$8 reported as optimum (Wang et al., 10 Jul 2025). DGF2 complements this process in two directions. Upstream, it structurally regularizes the student’s features before distillation. Downstream, as DGKD aligns the semantic content of the student to the teacher, DGF2 helps retain object contours and edges after alignment.

This coupling between geometric priors and semantic transfer is one of the distinctive claims of DGKD-WLSS. The paper positions the combination as directly targeting low-light-specific issues—CAM unreliability and semantic confusion—under weak supervision, rather than merely improving generic RGB-D fusion (Wang et al., 10 Jul 2025).

5. Empirical profile, efficiency, and failure modes

The reported ablations isolate DGF2’s contribution within DGKD-WLSS (Wang et al., 10 Jul 2025). On synthetic dark VOC, the baseline student achieves mIoU 43.4 and PixAcc 81.1; adding DGKD yields mIoU 55.2 and PixAcc 87.4; adding DGKD + DGF2 (multi-stage) yields mIoU 57.1 and PixAcc 87.9. On the real LIS dataset, the baseline student achieves mIoU 43.9 and PixAcc 78.1; adding DGKD yields mIoU 52.2 and PixAcc 87.2; adding DGKD + DGF2 yields mIoU 54.1 and PixAcc 86.5.

Setting	mIoU	PixAcc
Dark VOC baseline student	43.4	81.1
Dark VOC +DGKD	55.2	87.4
Dark VOC +DGKD + DGF2 (multi-stage)	57.1	87.9
LIS baseline student	43.9	78.1
LIS +DGKD	52.2	87.2
LIS +DGKD + DGF2	54.1	86.5

The paper also separates the effect of SFT from the effect of the consistency-aware gate and from hierarchical insertion. On dark VOC with DGKD, Single SFT reaches 56.1 mIoU and 87.8 PixAcc; Single DGF2 reaches 56.3 and 87.8; Multiple SFT reaches 56.7 and 87.8; Multiple DGF2 reaches 57.1 and 87.9. The reported conclusion is that the attention-gated consistency in DGF2 plus hierarchical fusion yields the best boundary and structure recovery.

Depth quality matters but weaker depth is still useful. With DGKD on dark VOC, Depth Anything reaches 57.1 mIoU and 87.9 PixAcc, ZoeDepth reaches 55.7 and 87.5, and MiDaS reaches 56.0 and 87.7. The paper explicitly states that even weaker or noisier depth estimates help, but quality matters.

The computational overhead of DGF2 is small relative to DGKD. At input size $F_d$9, the baseline student has 138.0M parameters and 277.0G FLOPs; +DGKD has 148.0M parameters and 296.7G FLOPs; +DGKD + DGF2 has 148.3M parameters and 297.9G FLOPs. The incremental cost of DGF2 over DGKD is therefore 0.3M parameters and 1.2G FLOPs. Diffusion adds six forward passes per batch during training and is not used at inference; DGF2 does not alter inference-time modularity and is described as having negligible runtime impact relative to DGKD.

Failure modes are correspondingly structural. If depth estimation is noisy, particularly on reflective surfaces, transparency, or textureless regions, DGF2’s priors may be less helpful or may introduce background bias. Fine-grained structures remain difficult, with reported examples including bus windows versus cars and incomplete segmentation of thin structures such as chair legs. The paper notes that integrating image enhancement upstream could further help (Wang et al., 10 Jul 2025).

6. Alternative realizations across depth completion, RGB-D perception, and multimodal fusion

Outside DGKD-WLSS, DGF2-like mechanisms differ substantially in implementation but not in objective. In "Learning Guided Convolutional Network for Depth Completion" (Tang et al., 2019), the fusion target is sparse depth completion rather than segmentation. RGB guidance features produce spatially-variant kernels that convolve depth features, and a two-stage factorization makes multi-stage guided fusion feasible. The reported KITTI validation ablation shows Add. at RMSE 809.37 mm, Concat. at 802.13 mm, and the multi-stage guided method at 777.78 mm, with the paper arguing that guided convolution improves over naïve addition or concatenation.

In "DFuseNet: Deep Fusion of RGB and Sparse Depth Information for Image Guided Dense Depth Completion" (Shivakumar et al., 2019), the fusion mechanism is far simpler: separate RGB and depth encoders, late concatenation of modality-specific SPP features, and multi-scale decoder aggregation. The paper does not use attention or gating. Its DGF2 relevance is therefore structural rather than terminological: depth guidance occurs through dual-branch late fusion after modality-specific processing.

In "Deep feature selection-and-fusion for RGB-D semantic segmentation" (Su et al., 2021), the paper explicitly states that it does not use the term “DGF2,” but its Symmetric Cross-modality Residual Fusion block instantiates explicit stage-wise RGB–depth interaction. At each encoder stage, depth features select complementary signals for RGB and vice versa through learned $F_d$0 projections and residual merging, while Detailed Feature Propagation reintroduces detail-rich fused features into the decoder. On NYUDv2 with a ResNet-101 backbone, the baseline element-wise sum fusion reaches 47.9% mIoU, +DFP reaches 49.6%, +SCRF reaches 50.8%, and +SCRF + DFP reaches 52.0%.

In "Hybrid Transformer Based Feature Fusion for Self-Supervised Monocular Depth Estimation" (Tomar et al., 2022), the paper describes a DGF2-like mask-guided multi-stream fusion rather than an explicit depth-input module. Two CNN encoders and a transformer encoder are fused at multiple scales; learned spatial masks gate the local CNN streams before multi-dilation fusion. The reported ablation progresses from Net1 (TG only) with RMSE 4.917, to Net3 (TG + LRL + HRL, atrous fusion) with RMSE 4.602, to Net5 (final: atrous fusion + masks) with RMSE 4.528, while Net4 shows that replacing the proposed fusion with simple sum degrades performance to RMSE 4.641.

In "Data-Level Recombination and Lightweight Fusion Scheme for RGB-D Salient Object Detection" (Wang et al., 2020), depth guidance is moved even earlier. The method cyclically replaces one RGB channel with depth to form three 3-channel inputs, $F_d$1, $F_d$2, and $F_d$3, which are then processed by three VGG16 backbones and fused through sparse cross-stream, multi-level connections. The full model has 91.8M parameters, 15.6 FPS at $F_d$4, and on NLPR improves from tri-stream lightweight concatenation at maxF 0.862 and MAE 0.038 to the proposed fusion at maxF 0.872 and MAE 0.035, with a recurrent refinement reaching maxF 0.880 and MAE 0.032. Here depth guidance is embedded at the data level rather than introduced as an explicit feature-side gate.

These examples show that DGF2 should not be reduced to one operator family. The surveyed implementations span dynamic kernels, late concatenation, residual cross-modality fusion, mask-guided transformer fusion, and pre-extraction channel recombination.

7. Temporal, sensor-aware, and language-conditioned extensions

Recent work extends DGF2 beyond static RGB-D fusion. In "CurriFlow: Curriculum-Guided Depth Fusion with Optical Flow-Based Temporal Alignment for 3D Semantic Scene Completion" (Lin et al., 14 Oct 2025), depth guidance is distributed across time, volumetric fusion, and 2D-to-3D lifting. Training uses

$F_d$5

with $F_d$6 described as a decaying weight function, and depth volumes are fused by confidence-gated 3D attention before Lift-Splat-Shoot voxelization and deformable attention. On SemanticKITTI validation, CurriFlow reports IoU 45.5 and mIoU 16.9; ablations show a progression from a 15.87 mIoU baseline to 16.66 with curriculum-guided depth fusion, 16.74 with CGAttention3D, and 16.89 with Grounded-SAM.

In "DGFusion: Depth-Guided Sensor Fusion for Robust Semantic Perception" (Broedermannn et al., 11 Sep 2025), DGF2 is defined as the feature-fusion block inside a multimodal semantic and panoptic perception framework. The module augments local-window cross-attention with a local depth token $F_d$7 and a global condition token $F_d$8. Queries are formed from $F_d$9, keys and values from a secondary modality, and the resulting attention acts as a soft, spatially varying, reliability-aware gate. The same work introduces a robust depth loss based on log-scale $F_{fuse}$0, $F_{fuse}$1-quantile filtering, image-edge-aware smoothness, and panoptic-edge-aware smoothness. On MUSES panoptic segmentation, DGFusion reports PQ 61.03 versus CAFuser 59.70, with gains of +1.34 in Fog, +1.63 in Rain, +2.57 in Snow, and +0.80 in Clear; on semantic segmentation it reports mIoU 79.5 versus CAFuser 78.2.

In "Language-Depth Navigated Thermal and Visible Image Fusion" (Zhang et al., 11 Mar 2025), DGF2 is realized without a single named fusion block. A diffusion U-Net is trained on concatenated visible and infrared inputs, CLIP text embeddings from depth-enriched captions are mapped to affine modulation parameters, and two auxiliary depth estimators supervise the fused image through scale-invariant log loss. The feature modulation is

$F_{fuse}$2

On the TNO dataset, the reported full model reaches SF 13.3423, Qab/f 0.6213, MI 16.8924, SD 92.1103, and VIF 0.9012. This formulation is notable because depth guides fusion both explicitly, through auxiliary depth losses on the fused image, and implicitly, through language-mediated semantic conditioning.

A broader implication of these extensions is that DGF2 has evolved from RGB-D feature mixing toward reliability-aware fusion in which depth informs temporal alignment, sensor trust calibration, volumetric lifting, and even semantic conditioning. The constant element is not a fixed block, but the use of geometry to decide what should be fused, where, and with what confidence.

8. Limitations, misconceptions, and open questions

A persistent limitation across the cited systems is dependence on depth quality. DGKD-WLSS reports degraded performance when depth estimation is noisy on reflective surfaces, transparency, or textureless regions (Wang et al., 10 Jul 2025). CurriFlow identifies stereo noise, sparsity at distance, and optical-flow errors under large motion or low texture as unresolved failure sources (Lin et al., 14 Oct 2025). DGFusion notes sensitivity to LiDAR quality, calibration, and unseen condition regimes despite its robust loss design (Broedermannn et al., 11 Sep 2025). The thermal-visible framework further depends on the quality of depth-enriched captions and on LiDAR-supervised depth pretraining on KAIST (Zhang et al., 11 Mar 2025).

Another misconception is that DGF2 always requires direct depth sensing at inference. The surveyed literature shows several alternatives. Some methods use projected LiDAR as both input and supervision, as in DGFusion (Broedermannn et al., 11 Sep 2025); some rely on monocular depth estimators such as Depth Anything, ZoeDepth, or MiDaS to produce $F_{fuse}$3 from RGB (Wang et al., 10 Jul 2025); some use stereo or completed LiDAR depth during training to stabilize optimization, then run in camera-only settings (Lin et al., 14 Oct 2025); and some use depth mainly as an auxiliary supervisory signal or as a latent semantic prior rather than as a fused branch in the final predictor (Zhang et al., 11 Mar 2025).

The main unresolved design question is not whether depth should be fused, but how explicitly and at what level of abstraction. The literature contains stage-wise feature modulation, dynamic filtering, residual fusion, local-token conditioning, 2D-to-3D lifting, and data-level recombination. This variety suggests that DGF2 is an architectural principle whose precise realization depends on task constraints: low-light segmentation emphasizes boundary recovery under weak supervision; depth completion emphasizes edge-aligned regression under sparsity; sensor fusion emphasizes reliability under adverse conditions; and thermal-visible fusion emphasizes geometry-consistent image synthesis.