DepthP+P: Fusion of Metric Priors and Predictions

• The paper integrates sparse metric priors with dense monocular depth predictions using localized affine transformations to achieve metrically aligned depth maps.
• DepthP+P employs a coarse-to-fine conditioned pipeline where pre-filled priors guide monocular depth estimation, improving performance on benchmarks like NYUv2 and ARKitScenes.
• The approach demonstrates robust zero-shot generalization and mixed-prior resilience by fusing planar, parallax, and prior-regularized mechanisms for diverse depth tasks.

DepthP+P refers to a collection of depth estimation methodologies that integrate planar modeling and parallax geometry with priors and predictions, with notable instantiations in both depth completion and monocular depth estimation. The key paradigm underlying DepthP+P is the combination of metric priors (e.g., sparse LiDAR or measured depth) and dense but potentially scale-ambiguous image-based predictions, fused through planar, parallax, and prior-regularized mechanisms to yield metrically accurate, dense depth maps across diverse scenarios. Recent frameworks under the DepthP+P banner include coarse-to-fine metric prior filling conditioned on monocular depth prediction, robust planar-parallax view synthesis for scale-disambiguated monocular learning, and hybrid optimization pipelines that exploit both spatial priors and learned geometric structure.

1. Pixel-Level Metric Alignment and Distance-Aware Fusion

DepthP+P frameworks exploit the complementary strengths of metric priors and image-based depth predictions. The prior, denoted $\mathbf D_{\rm prior}$, provides accurate but incomplete metric measurements (e.g., from LiDAR, SfM, or low-resolution capture), available only on a sparse set $\mathcal P = \{(x_i, y_i)\}_{i=1}^N$. A monocular depth prediction, $\mathbf D_{\rm pred}$, offers a dense but relative (non-metric) geometric structure.

The goal is to pre-fill missing pixels in $\mathbf D_{\rm prior}$ by aligning and fusing $\mathbf D_{\rm pred}$ using a localized affine transformation. For each unobserved pixel $(\hat x, \hat y)$:

• $K$ nearest neighbors in $\mathcal P$ are selected.
• Local linear parameters (scale $s$, shift $t$) are fitted by solving

$\mathcal P = \{(x_i, y_i)\}_{i=1}^N$0

with distance-aware weights $\mathcal P = \{(x_i, y_i)\}_{i=1}^N$1.

• The new value is filled as $\mathcal P = \{(x_i, y_i)\}_{i=1}^N$2.

This procedure enforces exact agreement at known prior locations, adapts local geometric cues, and produces dense, metrically aligned priors as input for subsequent refinement steps (Wang et al., 15 May 2025).

2. Conditioned Monocular Depth Estimation and Coarse-to-Fine Pipeline

After local metric alignment, $\mathcal P = \{(x_i, y_i)\}_{i=1}^N$3 serves as a dense but noisy provisional estimate. A lightweight conditioned MDE model receives:

• The RGB image $\mathcal P = \{(x_i, y_i)\}_{i=1}^N$4,
• The normalized pre-filled prior $\mathcal P = \{(x_i, y_i)\}_{i=1}^N$5,
• The normalized prediction $\mathcal P = \{(x_i, y_i)\}_{i=1}^N$6.

These are embedded via zero-initialized convolutions and concatenated with image features before being decoded to a final dense metric prediction $\mathcal P = \{(x_i, y_i)\}_{i=1}^N$7.

The system is trained with a per-pixel $\mathcal P = \{(x_i, y_i)\}_{i=1}^N$8 loss against metric ground truth: $\mathcal P = \{(x_i, y_i)\}_{i=1}^N$9 This pipeline enables the effective fusion of both accurate priors and complete geometric predictions and is robust to varying prior sparsity or pattern (Wang et al., 15 May 2025).

3. Zero-Shot Generalization and Task Diversity

DepthP+P demonstrates strong zero-shot generalization across depth completion, super-resolution, and inpainting tasks without task-specific retraining:

• Depth completion with sparse priors (e.g., SfM or 8 LiDAR lines),
• Super-resolution from low-res priors (captured or $\mathbf D_{\rm pred}$0, $\mathbf D_{\rm pred}$1 synthetically downsampled),
• Inpainting (out-of-range, random square, object masks),
• Mixed priors combining several patterns.

On benchmarks such as NYUv2, ScanNet, ETH-3D, DIODE, KITTI, ARKitScenes, and RGB-D-D, DepthP+P (e.g. PriorDA DAv2-B+ViT-B) matches or surpasses prior task-specific methods in Absolute Relative Error (AbsRel), achieving, for example, AbsRel = 2.38 on NYUv2 completion (better than Marigold-DC at 2.65) and 1.94 on ARKitScenes super-resolution (second best overall), while also providing sharp boundary preservation and robustness to novel prior patterns (Wang et al., 15 May 2025).

4. Mixed-Prior Robustness and Test-Time Model Adaptivity

One of the central advantages of the DepthP+P approach is its resilience to mixed or previously unseen prior types, achieved through robust local alignment of priors and implicit model regularization. When tested on combinations of sparse, low-res, and masked priors, error increases are minimal (e.g., AbsRel on NYUv2 rises only 0.05 over single-task), in contrast to prior-specific approaches which degrade substantially under such settings.

Furthermore, DepthP+P allows for test-time swapping of either the frozen monocular depth predictor or the conditioned MDE backbone without retraining, enabling an application-dependent trade-off between accuracy and inference speed. The approach can leverage advances in monocular depth estimation models or lower the computational footprint as needed (Wang et al., 15 May 2025).

5. Theoretical Context: Planar, Parallax, and Prior Integrations

The term DepthP+P has also been used in the literature for methods drawing on the classical plane+parallax (P+P) paradigm (Safadoust et al., 2023, Liang et al., 2023). In monocular settings, plane-induced homographies aligned to a known ground plane decouple rotation from translation, reducing self-supervised depth estimation to a metric scale by leveraging a single known parameter (e.g., camera height). The residual parallax is then used to estimate depth and translation, avoiding the scale ambiguity of full 6DoF methods: $\mathbf D_{\rm pred}$2 The DepthP+P paradigm via this geometric decoupling enables metrically accurate monocular estimation without the need for post-hoc scaling or ground-plane fitting at test time. Extended versions, such as PPNet, introduce flow-pretraining and planar position embeddings to handle dynamic scenes, sloped roads, and occlusion boundaries. These variants consistently demonstrate improved metric accuracy and boundary localization (Safadoust et al., 2023, Liang et al., 2023).

6. Comparison with Related Prior+Prediction Fusion Frameworks

DepthP+P frameworks differ fundamentally from earlier plug-and-play or posterior regularization approaches (Wang et al., 2018, Yang et al., 2019) in several respects:

• Plug-and-Play modules (PnP) update intermediate network features via sparse depth constraints without retraining, but optimize only through feature-space gradients and cannot introduce new geometric structure.
• Dense Depth Posterior methods realize full Bayesian posteriors as products of learned image-conditioned priors and sparse-observation likelihoods, suitable for uncertainty quantification but computationally more involved.
• Piecewise-planar CRF methods (Zhong et al., 2020) enforce hard planar regularization via superpixel decomposition and CRF inference, achieving sharp boundaries, yet lack flexible fusion with arbitrary prior formats or dense predictions.

DepthP+P distinguishes itself by merging arbitrary priors with dense predictions in a spatially adaptive, learnable, and scalable manner, robust to diverse prior types and patterns, and directly yielding metrically accurate depth maps (Wang et al., 15 May 2025).

7. Limitations and Directions for Future Research

Despite the demonstrated flexibility and generalization of DepthP+P, current limitations include:

• Sensitivity to prior noise if input priors contain outliers or systematic error, though the conditioned MDE refines but cannot fully denoise extreme outliers.
• Dependence on the spatial support and distribution of prior observations; extremely sparse or ill-distributed priors may yield local artifacts.
• The need for high-quality monocular depth predictors as inputs; improvements in backbone MDE architectures directly translate to DepthP+P performance.
• For P+P geometric methods, reliance on ground-plane detection and static-scene assumptions; dynamic or non-planar scenarios challenge the theory and empirical accuracy.

Future directions include automated, learned prior selection, multi-plane or layered parallax handling for highly non-planar scenes, and principled confidence modeling for downstream 3D perception tasks (Wang et al., 15 May 2025, Safadoust et al., 2023, Liang et al., 2023).