Depth-Guided Network (DGN)

• DGN is a design pattern where depth explicitly guides neural computations, modulating feature extraction and fusion rather than serving as incidental metadata.
• Architectural implementations vary from early RGB-D fusion and dual-branch designs to bidirectional query injection, each balancing efficiency with enhanced task performance.
• Empirical findings demonstrate that effective depth guidance can improve image restoration, grasp detection, and robotic performance, though its impact is sensitive to depth quality.

Taken across the cited literature, Depth-Guided Network (DGN) is best understood as an umbrella designation for neural architectures in which depth acts as an explicit guiding modality for another prediction problem, rather than as incidental metadata or a purely downstream cue. In some works the term is explicit, as in the dual-branch restoration model called DGN (He et al., 10 Aug 2025); in others it is the functional role of the architecture, as in depth-guided relighting, video recognition, grasp detection, depth completion, saliency detection, and underwater enhancement (Yang et al., 2021, Fu et al., 2020, Qin et al., 2023, Gu et al., 2021, Huang et al., 2024). A plausible implication is that DGN is less a single canonical network than a recurring design pattern: depth may be fused at the input, encoded in a separate branch, used to predict affine modulation parameters, injected through cross-modal attention, converted into learned affinities for a variational solver, or transformed into masks that route local and non-local processing.

1. Conceptual scope and taxonomy

The literature supports a broad but technically coherent definition. A DGN uses depth to guide feature extraction, feature fusion, normalization, attention, residual prediction, or supervision. This distinguishes it from systems in which depth is only the final regression target, only a post-processing signal, or only a label source. The strongest examples are those in which depth changes the internal computation of the network itself, such as S3Net’s 8-channel RGB-D early fusion for relighting, AMeFu-Net’s depth-conditioned adaptive instance normalization, DGCAN’s RGB-guided refinement of noisy depth features, DenseLiDAR’s pseudo-depth-guided residual completion, UVZ’s depth-conditioned local/non-local enhancement, and the 2025 DGN’s bidirectional image-depth interaction through its Depth-guided Spatial Enhancement module (Yang et al., 2021, Fu et al., 2020, Qin et al., 2023, Gu et al., 2021, Huang et al., 2024, He et al., 10 Aug 2025).

Some papers are closely related but not strict instances of this pattern. SDNet is explicitly described as “semantically guided depth estimation,” yet its mechanism is shared multi-task representation learning rather than an explicit guidance block; semantics help depth through a shared encoder/ASPP and joint loss, not by dynamically modulating the depth branch (Ochs et al., 2019). Likewise, EGD-Net is an edge-guided monocular depth estimator: its auxiliary branch uses image gradients and edge supervision to guide depth prediction, but the guidance signal is edge structure rather than depth (Dong et al., 2022).

Model	Task	Depth-guidance form
S3Net	Depth-guided image relighting	Early RGB-D fusion in a single encoder-decoder stream
MBNet	Depth-guided one-to-one relighting	Bifurcated RGB/depth encoder with DDPM fusion
AMeFu-Net	Few-shot video recognition	Depth-conditioned adaptive instance normalization
DGCAN	RGB-D planar grasp detection	Cross-modal attention and explicit grasp-depth regression
DenseLiDAR	Depth completion	Pseudo-depth-guided residual learning and structural loss
UVZ / DGEN	Underwater image enhancement	Predicted-depth-guided local/non-local perception
DGN	Image restoration	Interactive dual branches with DSE cross-branch exchange

This taxonomy also clarifies a common misconception. A DGN is not defined merely by the presence of an RGB-D input pair. PDNet makes this distinction explicit by arguing against treating depth as a fourth input channel and instead using an independent depth-enhanced subsidiary network whose output is incorporated into the RGB master network (Zhu et al., 2018). The same distinction reappears in AMeFu-Net, whose ablation shows that simple RGB-depth concatenation is markedly weaker than depth-guided modulation (Fu et al., 2020).

2. Recurrent architectural organizations

One recurrent organization is single-stream early fusion. S3Net states the formulation most explicitly: the source RGB image, source depth map, guided RGB image, and guided depth map are concatenated into an 8-channel tensor,

$$X = \operatorname{Concat}(I_s, D_s, I_g, D_g) \in \mathbb{R}^{H \times W \times 8},$$

and mapped to a 3-channel relit image through an encoder-decoder network with skip connections,

$\hat{I}_{relit} = F(I_s, D_s, I_g, D_g).$

Its encoder is Res2Net101, modified to accept 8 input channels, and its decoder contains an attention module and an enhanced module. The paper argues that this single-stream design is practical for $1024 \times 1024$ inputs and four aligned modalities, especially because relighting depends on global illumination direction and patchwise cropping is not appropriate (Yang et al., 2021).

A second organization is the bifurcated or dual-branch encoder. MBNet exemplifies this design for relighting by using two ResNet-50 backbones without weight sharing, one for RGB and one for depth, and fusing features from conv3, conv4, and conv5. Its decoder then uses Dynamic Dilated Pyramid Modules (DDPMs) to integrate texture and structure across multiple receptive fields (Yang et al., 2021). PDNet uses an analogous master-subsidiary decomposition: a VGG-based RGB saliency network is guided by an independent depth branch that acts as a depth-enhanced weight prediction network, rather than by naive RGB-D channel stacking (Zhu et al., 2018).

A third organization is the interactive mirrored dual branch. The 2025 DGN for image restoration uses a depth estimation branch and an image restoration branch with mirrored structures. Each residual group exchanges complementary features across branches through the Depth-guided Spatial Enhancement module, so depth guidance appears repeatedly through the hierarchy rather than only at the input or output (He et al., 10 Aug 2025). UVZ adopts a related two-stage decomposition: a depth estimation network first predicts depth, and a depth-guided enhancement network then parses near-far scenarios and routes processing through local and non-local branches (Huang et al., 2024).

A fourth organization is model-based guidance with unrolled optimization. The deep primal-dual network for guided depth super-resolution does not merely fuse depth and guidance with convolutions. An FCN predicts both an initial high-resolution depth estimate $d^{(\mathrm{hr})}$ and learned affinity terms $A$, and a non-local variational model is then solved by 20 unrolled primal-dual iterations. Guidance is therefore embedded both in the FCN input $s=(d^{(\mathrm{mr})}, g)$ and in the learned regularization weights of the optimization layer (Riegler et al., 2016).

3. Guidance operators and fusion mechanisms

The most explicit depth-guided modulation operator in the cited literature is AMeFu-Net’s Depth Guided Adaptive Instance Normalization (DGAdaIN). With RGB and depth feature sequences

$$\mathrm{I}_{rgb}, \mathrm{I}_d \in \mathbb{R}^{B \times D \times L},$$

the fused representation is

$$f\left(\mathrm{I}_{rgb},\mathrm{I}_{d}\right)=g_{s}\left(\mathrm{I}_{d}\right)\cdot\frac{\mathrm{I}_{rgb}-\mathrm{\mu}(\mathrm{I}_{rgb})}{\mathrm{\sigma}(\mathrm{I}_{rgb})}+g_{b}\left(\mathrm{I}_{d}\right),$$

where $g_s$ and $g_b$ are learnable fully connected layers that generate scale and bias from depth features. The asymmetry is deliberate: RGB remains the content-bearing stream, while depth provides contextual guidance (Fu et al., 2020).

Cross-modal attention is another major mechanism. DGCAN’s Local Cross-modal Attention module defines the fused feature as

$\hat{I}_{relit} = F(I_s, D_s, I_g, D_g).$0

with queries from RGB and keys/values from depth. For each position $\hat{I}_{relit} = F(I_s, D_s, I_g, D_g).$1, only a local neighborhood $\hat{I}_{relit} = F(I_s, D_s, I_g, D_g).$2 is attended: $\hat{I}_{relit} = F(I_s, D_s, I_g, D_g).$3 This asymmetry reflects the paper’s premise that RGB is the cleaner modality and should be used to refine noisy depth before fusion (Qin et al., 2023).

The 2025 DGN generalizes this idea to bidirectional cross-branch query injection. In its DSE module, image features $\hat{I}_{relit} = F(I_s, D_s, I_g, D_g).$4 are split, one half enters sparse non-local InterSIM, and the other half interacts with depth features $\hat{I}_{relit} = F(I_s, D_s, I_g, D_g).$5 through SSC and CSC blocks. The image branch forms

$\hat{I}_{relit} = F(I_s, D_s, I_g, D_g).$6

while the depth branch forms

$\hat{I}_{relit} = F(I_s, D_s, I_g, D_g).$7

This is not simple concatenation; each branch injects queries into the other branch’s spatial self-correlation computation (He et al., 10 Aug 2025).

Depth can also guide a network through residual baselines and explicit structural priors. DenseLiDAR constructs a dense pseudo-depth $\hat{I}_{relit} = F(I_s, D_s, I_g, D_g).$8 by morphology, rectifies sparse input against that guide, and predicts only the residual,

$\hat{I}_{relit} = F(I_s, D_s, I_g, D_g).$9

The guide therefore determines what the network must correct rather than forcing direct dense regression from sparse measurements alone (Gu et al., 2021).

In UVZ, depth guidance is transformed into branch-specific masks. The non-local branch of the Depth Perception Module computes

$1024 \times 1024$0

while the local branch applies

$1024 \times 1024$1

Here $1024 \times 1024$2 come from the paper’s R$1024 \times 1024$3N transformation, and the point of the design is to let near and far regions prefer different receptive fields (Huang et al., 2024).

A more classical but still highly technical guidance mechanism appears in the deep primal-dual network. The learned affinity term enters the non-local regularizer through

$1024 \times 1024$4

The guidance image thus affects reconstruction by controlling which pixels are allowed to smooth each other in the unrolled variational solver (Riegler et al., 2016).

4. Supervision strategies and optimization objectives

DGN-style models often pair architectural guidance with structure-aware supervision. S3Net uses a three-term objective,

$1024 \times 1024$5

with $1024 \times 1024$6, $1024 \times 1024$7, and $1024 \times 1024$8. The Charbonnier term stabilizes reconstruction, the wavelet-SSIM term enforces multiscale structural fidelity through discrete wavelet decomposition, and the VGG19 perceptual term promotes feature-level consistency. The paper explicitly uses no adversarial loss (Yang et al., 2021).

DenseLiDAR formalizes a different supervisory regime for sparse-label settings: $1024 \times 1024$9 with $d^{(\mathrm{hr})}$0 and

$d^{(\mathrm{hr})}$1

The noteworthy element is that $d^{(\mathrm{hr})}$2, a pseudo dense ground truth generated morphologically from sparse ground truth, is used only for dense structural supervision, while metric supervision remains tied to sparse valid depth values (Gu et al., 2021).

The 2025 DGN adopts a lighter objective and relies more heavily on architecture. Its total loss is

$d^{(\mathrm{hr})}$3

with

$d^{(\mathrm{hr})}$4

$d^{(\mathrm{hr})}$5

and $d^{(\mathrm{hr})}$6. The affine-invariant depth term is borrowed from Depth Anything and is intended to preserve relative depth structure rather than only absolute reconstruction (He et al., 10 Aug 2025).

UVZ separates training into two stages rather than optimizing a single joint loss. The first stage trains the depth estimator and auxiliary supervision network with

$d^{(\mathrm{hr})}$7

where $d^{(\mathrm{hr})}$8. The second stage trains the enhancement network with

$d^{(\mathrm{hr})}$9

where $A$0 and $A$1. This decoupling reflects the paper’s decision to pretrain depth guidance first and freeze it for enhancement (Huang et al., 2024).

Adjacent work clarifies what does not count as an explicit DGN loss design. SDNet’s total objective,

$A$2

with $A$3, is a shared multi-task learning objective. Semantics guide depth through shared features and joint supervision, but there is no separate semantic-to-depth modulation block (Ochs et al., 2019).

5. Empirical behavior across tasks

The strongest direct evidence for DGN behavior is usually found in ablations that compare depth-guided fusion against RGB-only or naive multimodal baselines. In S3Net, the “Image only” setting yields PSNR 18.7611 and SSIM 0.6821, adding depth with the same loss yields PSNR 18.8451 and SSIM 0.6913, and replacing plain SSIM with wavelet-SSIM yields PSNR 19.1281 and SSIM 0.6969. On the NTIRE 2021 benchmark, the method reports validation SSIM 0.7022 and PSNR 19.2462, and testing MPS 0.6452, SSIM 0.6784, LPIPS 0.1566, and PSNR 19.2212, corresponding to 3rd place in SSIM, 4th in PSNR, 2nd in LPIPS, and 3rd in MPS (Yang et al., 2021).

AMeFu-Net provides a particularly clean demonstration that how depth guides RGB matters. On Kinetics 5-way few-shot recognition, RGB + Depth + Concat gives 67.5 / 76.9 / 81.1 / 82.0 / 83.3 for 1–5 shot, whereas RGB + Depth + DGAdaIN gives 73.6 / 80.7 / 84.0 / 85.4 / 86.6. The full model with temporal asynchronization reaches 74.1 / 81.1 / 84.3 / 85.6 / 86.8. The direction-of-guidance ablation is even more decisive: “RGB guide Depth” gives 50.6 / 58.7 / 64.3 / 64.1 / 66.1, two-way guidance gives 66.5 / 74.8 / 78.3 / 79.4 / 82.8, and “Depth guide RGB” remains best at 73.6 / 80.7 / 84.0 / 85.4 / 86.6 (Fu et al., 2020).

DGCAN shows that depth guidance can improve not only benchmark scores but executable robotic behavior. On the GraspNet-Planar benchmark, DGCAN reaches 49.85/47.32 AP on seen objects, 41.46/35.73 on similar objects, and 17.48/16.10 on novel objects for RealSense/Kinect, improving further with collision detection. Its depth ablation shows explicit depth regression is better than using center depth or depth classification. On real robot evaluation, DGCAN achieves single-object GSR 88.00% (66/75), multi-object GSR 84.75% (50/59), and SCR 100.00% (50/50), outperforming Dex-Net 4.0 and FC-GQ-CNN (Qin et al., 2023).

DenseLiDAR is notable because it makes depth guidance central to a real-time system. On KITTI test it reports RMSE 755.41 mm, MAE 214.13 mm, iRMSE 2.25, iMAE 0.96, and 50 FPS. Its ablation from baseline to rectification, structural loss, and residual learning reduces RMSE from 829.87 to 795.97, RMSE$A$4 from 1596.22 to 1335.20, and RMSE$A$5 from 2794.98 to 2171.76, indicating that the pseudo-depth guide is especially beneficial for boundary geometry (Gu et al., 2021).

The 2025 DGN presents a more modest but consistent depth-branch gain in restoration. On common $A$6 SR benchmarks, “w/o Depth” reports 30.24 PSNR / 0.8324 SSIM / 0.2800 LPIPS, while “w Depth” reports 30.28 / 0.8335 / 0.2787. The model has 3.52M parameters and is particularly competitive in LPIPS, reporting, for example, 0.2495 on Urban100 and 0.2787 on DIV2K-VAL (He et al., 10 Aug 2025).

UVZ demonstrates that depth guidance remains useful even when depth is itself predicted. On the CYCLE dataset it reports PSNR 26.00, SSIM 0.8721, MSE 0.00435, UIQM 5.044, UICM 4.781, NIQE 6.586, and runtime 0.0583 s. In the second-stage ablation, removing depth maps from DPM lowers PSNR from 26.00 to 25.76 and UICM from 4.781 to 3.613, which is the clearest quantitative indication that the predicted depth prior is materially affecting enhancement quality (Huang et al., 2024).

6. Limitations, boundary cases, and research directions

The literature also defines the limits of DGN reasoning. In relighting, S3Net reports failure cases when the source image contains large shadow regions. Even with depth maps, the network struggles because the available depth encodes only visible front-side spatial structure rather than complete omnidirectional scene geometry; color temperature may still transfer correctly, but structures under heavy shadows can be poor (Yang et al., 2021). This suggests that a DGN is not a substitute for full physical scene understanding.

Depth quality itself is often the dominant bottleneck. DGCAN is motivated by the fact that consumer RGB-D sensors produce depth that is noisier and lower quality than RGB, which is why its fusion is deliberately asymmetric and RGB-guided (Qin et al., 2023). DenseLiDAR makes the same point differently: its pseudo-depth guide is dense and structurally informative, but not accurate enough to be a final output, so the system must correct it with residual learning and even use it again for post-hoc outlier rejection (Gu et al., 2021). The 2025 DGN similarly relies on low-quality depth generated by Depth Anything V2 rather than sensor ground truth, and its main ablation shows only modest numerical gains, even though they are consistent (He et al., 10 Aug 2025).

Another limitation is that not every “guided” network should be collapsed into the DGN label. SDNet is best read as a joint multi-task model in which semantic supervision regularizes shared features for depth estimation, not as an explicit guidance-block architecture (Ochs et al., 2019). EGD-Net is an edge-guided depth estimator, not a depth-guided network in the strict sense, because the auxiliary signal is image gradient and learned edge attention rather than depth (Dong et al., 2022). A plausible implication is that the term DGN is most precise when depth modulates another stream or solver directly.

The most explicit future direction in the cited literature comes from S3Net, whose authors suggest designing “a novel backbone to extract and fuse image and depth features more effectively” (Yang et al., 2021). Related work points toward the same trajectory from different angles: better bidirectional feature exchange in DSE (He et al., 10 Aug 2025), more robust attention under modality discrepancy in LCA (Qin et al., 2023), stronger structural priors under sparse supervision (Gu et al., 2021), and more selective routing between local and non-local processing in depth-conditioned enhancement (Huang et al., 2024). Taken together, the field indicates that the enduring technical question is not whether depth should be used, but how depth should govern computation: by concatenation, conditioning, affinity learning, structural masking, or task-specific residual parameterization.