PGDM-Loss: Prior-Guided Masking for Depth

• The paper introduces PGDM-Loss, a novel method using a dynamic mask based on reprojection error to discount unstable regions during training.
• It integrates cost volume auto-masking and spectral entropy priors to refine multi-frame depth estimation in dynamic scenes.
• Empirical results on benchmarks like KITTI and Cityscapes show improved depth accuracy with lower Absolute Relative errors and higher δ1 metrics.

Searching arXiv for the cited paper and closely related methods to ground the article with current identifiers. arXiv search query: (Chen et al., 2024) Prior-Guided Dynamic Mask Loss (PGDM-Loss) denotes a loss-design perspective for self-supervised depth estimation in dynamic scenes in which priors about pixel trustworthiness are used to mask or weight photometric supervision. In the formulation made explicit by D$^3$epth, the central prior is a reprojection-error-based dynamic mask that removes pixels with jointly high reprojection error across adjacent source frames from the photometric loss; this is complemented by an inter-frame appearance-consistency prior for cost-volume construction and a spectral-entropy prior for uncertainty-aware depth fusion (Chen et al., 2024). Within this framework, PGDM-Loss is not a separate network component but a way of organizing supervision so that regions inconsistent with the rigid-motion assumption contribute less, or not at all, to self-supervised training.

1. Formal setting within self-supervised monocular depth estimation

D$^3$epth is built directly on the standard self-supervised monocular scheme of Zhou et al. and Monodepth2, then modifies the loss with a dynamic mask (Chen et al., 2024). Each training sample is a triplet of consecutive monocular images,

$I_{t-1},\ I_t,\ I_{t+1},$

with $I_t$ as the target frame and $I_s$, $s\in\{t-1,t+1\}$, as source frames. Supervision is purely photometric via view synthesis, and no ground-truth depth is used.

The model employs a DepthNet and a PoseNet in a standard self-supervised pipeline. DepthNet contains a MonoDepth branch that predicts single-frame depth $D_{\text{Mono}}$, a MultiDepth branch that predicts multi-frame depth $D_{\text{Multi}}$ via cost volume, and a Spectral Entropy Uncertainty (SEU) module used to fuse the two predictions. The paper also follows a teacher–student distillation scheme from Xiang et al. [33], although the pose estimation network is not detailed in the provided description (Chen et al., 2024).

Target-view synthesis is defined as

$$I_{s \rightarrow t} = I_s\left\langle \text{Proj}(D_t,\ T_{t \rightarrow s},\ K) \right\rangle,$$

where $K$ is the camera intrinsics, $^3$0 projects pixels from the target frame into the source frame using depth and pose, and $^3$1 denotes differentiable sampling. The photometric error is

$^3$2

Following Monodepth2, the baseline photometric term is the per-pixel minimum reprojection loss

$^3$3

In the PGDM-Loss interpretation, this baseline is not discarded; rather, it is reweighted spatially by priors that identify where rigid-view synthesis is unreliable. The significance of the formulation is therefore not a change in the self-supervised paradigm itself, but a change in which pixels are permitted to define the photometric training signal.

2. Reprojection-error prior and dynamic-mask construction

The central prior in D$^3$4epth is that pixels with high reprojection error in both adjacent source views are likely to correspond to dynamic objects, occlusion regions, or strong photometric violations, which are precisely the locations where self-supervised geometry from rigid motion is unreliable (Chen et al., 2024). For each source frame $^3$5, the method computes a synthesized image and a per-pixel reprojection error: $^3$6

$^3$7

Stacking the two source-view errors yields

$^3$8

where the two channels correspond to $^3$9 and $I_{t-1},\ I_t,\ I_{t+1},$0. For each channel $I_{t-1},\ I_t,\ I_{t+1},$1, the spatial dimensions are flattened,

$I_{t-1},\ I_t,\ I_{t+1},$2

and a quantile threshold $I_{t-1},\ I_t,\ I_{t+1},$3 is computed,

$I_{t-1},\ I_t,\ I_{t+1},$4

with $I_{t-1},\ I_t,\ I_{t+1},$5 controlling the cutoff; the example given is $I_{t-1},\ I_t,\ I_{t+1},$6, corresponding to the top 20% highest reprojection errors per source view.

Binary high-loss masks are then defined as

$I_{t-1},\ I_t,\ I_{t+1},$7

The dynamic mask is

$I_{t-1},\ I_t,\ I_{t+1},$8

This mask is explicitly a keep-mask. A pixel is retained when at least one source view does not exhibit extremely high error, and it is excluded only when both source views are highly inconsistent. Pixelwise, $I_{t-1},\ I_t,\ I_{t+1},$9 denotes a dynamic, occluded, or otherwise outlier-like location as judged by the dual-view reprojection prior, while $I_t$0 indicates that the pixel remains in photometric supervision (Chen et al., 2024).

The thresholding strategy is adaptive rather than based on a fixed absolute error. This matters because it makes the mask depend on the current reprojection-error distribution in each batch item. A plausible implication is that the mask is less sensitive to scale shifts in photometric error than a fixed-threshold alternative, although the provided description does not report a direct comparison.

3. Loss integration and the PGDM-Loss formulation

The total objective in D$I_t$1epth is

$I_t$2

where each branch-specific loss takes the form

$I_t$3

The masked photometric term is written explicitly as

$I_t$4

The smoothness term is the standard edge-aware regularizer,

$I_t$5

A defining property of this PGDM-Loss instantiation is that the smoothness term is not masked. The paper states that it still regularizes the depth even in regions not supervised photometrically (Chen et al., 2024). Thus, the loss does not simply discard dynamic or unreliable regions altogether; it suppresses their direct photometric contribution while preserving regularization over the full field.

The practical effect is stated directly: high-error pixels in both source views are removed from photometric supervision, preventing them from pulling the network toward incorrect depths while still retaining gradients in well-behaved pixels (Chen et al., 2024). This also clarifies a common misunderstanding. The dynamic mask is not an additional prediction head and not an explainability mask learned from scratch; it is a binary gating variable derived from reprojection statistics.

From the more general PGDM-Loss perspective provided in the source, the same pattern can be abstracted as

$I_t$6

with $I_t$7 interpreted as a prior-guided weight. In D$I_t$8epth, the realized form is binary and driven primarily by reprojection-error quantiles (Chen et al., 2024).

4. Auxiliary priors beyond the loss-level dynamic mask

D$I_t$9epth supplements the loss-level dynamic mask with two additional priors: a cost volume auto-mask based on inter-frame appearance consistency, and a spectral-entropy prior over cost-volume reliability (Chen et al., 2024). These are not identical to the dynamic mask, but they belong to the same broader PGDM-Loss design logic in which priors determine which evidence should influence depth estimation.

For the multi-frame branch, features from the reference and source views are used to construct a cost volume over depth or disparity hypotheses. Before cost-volume construction, the method defines a pixelwise equality mask

$I_s$0

and then

$I_s$1

After downsampling according to feature scale,

$I_s$2

the mask is applied elementwise to feature maps,

$I_s$3

$I_s$4

The stated prior intuition is that pixels whose appearance is unchanged between frames carry no useful parallax, for example because the camera is stationary or the region is co-moving with the camera. The method therefore ignores them in cost-volume matching by zeroing the corresponding features (Chen et al., 2024). Importantly, this mask does not appear explicitly in the loss; it modifies the features that produce $I_s$5, and thereby influences the losses on $I_s$6 and $I_s$7.

The third prior is spectral entropy. Let $I_s$8 denote a cost probability tensor. The SEU module applies a Fourier transform,

$I_s$9

computes the magnitude spectrum $s\in\{t-1,t+1\}$0, normalizes it into a distribution $s\in\{t-1,t+1\}$1, and defines spectral entropy as

$s\in\{t-1,t+1\}$2

A small neural network maps $s\in\{t-1,t+1\}$3 to an uncertainty map $s\in\{t-1,t+1\}$4, which is then used for depth fusion: $s\in\{t-1,t+1\}$5

The interpretation given in the source is that high spectral entropy indicates a more complex or noisy pattern and hence higher uncertainty, whereas low spectral entropy indicates a more structured signal and lower uncertainty (Chen et al., 2024). This module is trained implicitly: there is no explicit loss directly on $s\in\{t-1,t+1\}$6 or $s\in\{t-1,t+1\}$7; instead, gradients pass through the fused depth term because $s\in\{t-1,t+1\}$8 is supervised by the same masked photometric-plus-smoothness loss.

5. Generalization as a prior-guided masking framework

The source explicitly relates D$s\in\{t-1,t+1\}$9epth to the notion of a “Prior-Guided Dynamic Mask Loss” and identifies three priors: a photometric reprojection error prior, an inter-frame appearance consistency prior for the cost volume, and a spectral entropy prior over cost-volume reliability (Chen et al., 2024). All three are described as priors on which pixels and which depth hypotheses are trustworthy, implemented as masks or weights inside the training objective.

Under this interpretation, a generalized PGDM-Loss can combine multiple prior-guided factors,

$D_{\text{Mono}}$0

where the reprojection prior may follow D$D_{\text{Mono}}$1epth’s binary dual-view rule, the appearance prior may down-weight pixels with unchanged raw RGB values, and the spectral prior may modulate confidence in multi-frame supervision through spectral entropy. The source also notes possible alternatives, such as soft masks, learned thresholds, temporal smoothing of masks, and spatial regularization, but presents these as generalization directions rather than as implemented components of D$D_{\text{Mono}}$2epth (Chen et al., 2024).

Several design axes are made explicit. First, there is a binary-versus-soft choice. D$D_{\text{Mono}}$3epth uses a binary dynamic mask $D_{\text{Mono}}$4 and a binary cost-volume auto-mask. The source notes that a PGDM-Loss could instead use soft masks, for example via a sigmoid of normalized reprojection error, in order to down-weight rather than fully ignore suspect pixels. Second, there is a static-versus-adaptive threshold choice. D$D_{\text{Mono}}$5epth uses quantile-based adaptive thresholds $D_{\text{Mono}}$6, which depend on the current error distribution. Third, there is a consistency choice: the implemented method handles each training sample independently, whereas a more advanced PGDM-Loss could impose temporal or spatial regularity on the mask.

The source also situates D$D_{\text{Mono}}$7epth relative to existing strategies. Monodepth2 uses minimum reprojection plus auto-masking to handle occlusions and static pixels, but can fail in dynamic scenarios where both reprojection errors are high. D$D_{\text{Mono}}$8epth adds the dynamic mask as a complement: when both views are bad, the contribution is suppressed entirely (Chen et al., 2024). The source further contrasts this with explainability masks in Zhou et al., which are learned mask networks; D$D_{\text{Mono}}$9epth instead uses hand-crafted priors based on reprojection, appearance consistency, and spectral entropy.

6. Empirical behavior, implementation details, and limitations

The implementation details reported for D$D_{\text{Multi}}$0epth are directly relevant to the PGDM-Loss interpretation. The architecture uses a DepthNet with a standard encoder–decoder backbone from Xiang et al. [33]. The multi-frame branch uses a cost volume constructed between $D_{\text{Multi}}$1 and an adjacent frame, with features masked by the cost-volume auto-mask. The SEU module is a small network that takes the spectral entropy map as input and outputs the uncertainty map $D_{\text{Multi}}$2 (Chen et al., 2024).

For the dynamic mask, the reported hyperparameter is $D_{\text{Multi}}$3, corresponding to the top 20% highest reprojection errors per source view. On KITTI, the dynamic mask is applied from epoch 0; on Cityscapes, it is applied from epoch 1 to allow initial stabilization. Training uses Adam with learning rate $D_{\text{Multi}}$4 for the teacher and $D_{\text{Multi}}$5 for the student, with a 10$D_{\text{Multi}}$6 decay schedule, batch size 12, and 20 epochs on KITTI and 5 epochs on Cityscapes (Chen et al., 2024).

The empirical results support the claim that the dynamic-mask prior is most consequential in scenes rich in dynamics. On KITTI, dynamic objects are stated to be rare, approximately $D_{\text{Multi}}$7 of pixels, so improvements are modest but still SOTA or near-SOTA. On Cityscapes, D$D_{\text{Multi}}$8epth reduces AbsRel to $D_{\text{Multi}}$9 and raises $$I_{s \rightarrow t} = I_s\left\langle \text{Proj}(D_t,\ T_{t \rightarrow s},\ K) \right\rangle,$$0 to $$I_{s \rightarrow t} = I_s\left\langle \text{Proj}(D_t,\ T_{t \rightarrow s},\ K) \right\rangle,$$1, outperforming several dynamic-aware baselines (Chen et al., 2024).

The ablation study on Cityscapes isolates the contribution of the individual priors. For single-frame inference, the baseline is AbsRel $$I_{s \rightarrow t} = I_s\left\langle \text{Proj}(D_t,\ T_{t \rightarrow s},\ K) \right\rangle,$$2, $$I_{s \rightarrow t} = I_s\left\langle \text{Proj}(D_t,\ T_{t \rightarrow s},\ K) \right\rangle,$$3 $$I_{s \rightarrow t} = I_s\left\langle \text{Proj}(D_t,\ T_{t \rightarrow s},\ K) \right\rangle,$$4; adding DM yields AbsRel $$I_{s \rightarrow t} = I_s\left\langle \text{Proj}(D_t,\ T_{t \rightarrow s},\ K) \right\rangle,$$5, $$I_{s \rightarrow t} = I_s\left\langle \text{Proj}(D_t,\ T_{t \rightarrow s},\ K) \right\rangle,$$6 $$I_{s \rightarrow t} = I_s\left\langle \text{Proj}(D_t,\ T_{t \rightarrow s},\ K) \right\rangle,$$7; adding CAM yields AbsRel $$I_{s \rightarrow t} = I_s\left\langle \text{Proj}(D_t,\ T_{t \rightarrow s},\ K) \right\rangle,$$8; adding SEU yields AbsRel $$I_{s \rightarrow t} = I_s\left\langle \text{Proj}(D_t,\ T_{t \rightarrow s},\ K) \right\rangle,$$9; and the full single-frame D$K$0epth model yields AbsRel $K$1, $K$2 $K$3. For multi-frame inference, the baseline is AbsRel $K$4, $K$5 $K$6; adding DM yields AbsRel $K$7, $K$8 $K$9; adding CAM yields AbsRel $^3$00; adding SEU yields AbsRel $^3$01; combining DM and CAM yields AbsRel $^3$02, $^3$03 $^3$04; and the full multi-frame D$^3$05epth model yields AbsRel $^3$06, $^3$07 $^3$08, the best result in the reported table (Chen et al., 2024). The FFT ablation further reports AbsRel $^3$09, $^3$10 $^3$11 without FFT and AbsRel $^3$12, $^3$13 $^3$14 with FFT.

The advantages reported by the authors are that the dynamic mask provides a simple but robust prior-guided mask based purely on reprojection-error statistics without extra networks, and that CAM and SEU incorporate multi-frame and frequency-domain priors without inference-time overhead (Chen et al., 2024). The limitations are equally explicit: high reprojection error is not uniquely caused by dynamics, since textureless areas, strong illumination changes, and motion blur can also produce high loss; the dynamic mask may therefore suppress some static regions that are merely difficult to reconstruct. The authors state that future work will focus on better distinguishing high-loss regions caused by dynamics from inherently hard regions.

These observations define the scope of PGDM-Loss as exemplified by D$^3$15epth. Its operative principle is not that every high-error region is dynamic, but that self-supervised photometric learning in dynamic scenes benefits when such regions are treated as unreliable unless corroborated by stronger geometric or appearance-based priors (Chen et al., 2024).