Plane-Residual Depth Completion

• The paper demonstrates that decomposing depth into an anchor plane and a residual simplifies the regression task, yielding improved RMSE performance on benchmarks.
• Plane-residual methods combine deep learning and analytic geometry to achieve real-time, interpretable, and efficient dense depth mapping for applications like KITTI and NYU-v2.
• This approach effectively tackles outlier rejection and robustness, though its reliance on global anchor planes may limit accuracy in highly non-planar scenes.

Plane-residual depth completion encompasses a family of methods for densifying sparse depth measurements (typically from LiDAR or structured light sensors) into high-resolution, pixel-dense depth maps by leveraging a two-stage "plane plus residual" representation. The unifying core of these approaches is the decomposition of per-pixel depth into a coarse "anchor" plane estimate—drawn either from learned discrete bins or from analytic local geometry—plus a fine residual correction. This paradigm is motivated by the observation that most real-world surfaces are locally planar or near-planar over the spatial support between sparse samples, and thus the full regression task can be simplified by partitioning large-scale structure (planes) from small-scale detail (residuals). Plane-residual methods can be instantiated using either deep learning architectures or analytic geometry-based pipelines, offering interpretable priors, bounded error, and efficient computation, and achieving strong performance on large benchmarks such as KITTI and NYU-v2 depth completion datasets (Lee et al., 2021, Zhao et al., 2021).

1. Plane-Residual Representation and Principle

The plane-residual (PR) representation expresses depth $d$ at each pixel $x$ as

$d(x) = d_{p(x)} + r(x),$

where $d_{p(x)}$ is the depth of the assigned anchor plane (with index $p$) and $r(x)$ is the residual offset from that plane. The set of anchor planes $\{d_1, d_2, \dots, d_D\}$ can be defined in two ways:

• Uniform & Relative (UR): The planes span the range of observed sparse depths, uniformly placed between the minimum and maximum values.
• Uniform & Absolute (UA): The planes are defined over a canonical metric interval (e.g., $[0,10]$ m for NYU-v2), with fixed absolute spacing.

Given a ground-truth depth $d^*(x)$, assignment proceeds by finding the nearest plane index $p^*$ and the corresponding residual $x$0, which by construction is bounded by half the plane spacing $x$1 (Lee et al., 2021).

Geometric variants instantiate the anchor plane not as a global bin, but as a local 3D plane fitted through the nearest sparse measurement and its neighbors. Empty pixels are populated by analytic extension: their depth is computed as the depth of the nearest LiDAR point plus a closed-form residual $x$2 determined by camera intrinsics, pixel offset, and estimated surface normal (Zhao et al., 2021).

2. Learning-Based Plane-Residual Architectures

The PR approach for deep networks employs an encoder–decoder backbone with a bifurcated head:

• Shared Encoder: ResNet-34 pretrained on ImageNet, receiving a 5-channel input (RGB, sparse depth, mask). Features are extracted at four spatial scales (1/4, 1/8, 1/16, 1/32).
• Dual Decoder Heads:
  • Plane-classification decoder ("P"): Outputs a $x$3-channel volume per pixel (softmax across planes). Trained with multi-class cross-entropy loss.
  • Residual-regression decoder ("R"): Outputs a single-channel map, with tanh activation scaled by $x$4 to ensure residuals remain within the correct metric support. Trained using an $x$5 regression loss.

The total loss is a balanced sum,

$x$6

with $x$7 in all reported experiments. Training uses canonical data augmentations and Adam optimizer settings. On both the KITTI and NYU-v2 datasets, this results in state-of-the-art speed-accuracy trade-offs, e.g., $x$812 ms per 640×480 image (83 fps) with a model size of 4.2M parameters (Lee et al., 2021).

3. Analytic Plane-Residual Methods for LiDAR Depth Completion

Non-learning ("label-free") PR methods for depth completion, exemplified by the pipeline in (Zhao et al., 2021), exploit the physical geometry of sparse point clouds:

• Local Plane Hypothesis: Each empty pixel is assumed to reside on the plane passing through its nearest valid LiDAR sample.
• Closed-Form Residual Calculation:
  • The nearest point $x$9, its depth $d(x) = d_{p(x)} + r(x),$0, and its local surface normal $d(x) = d_{p(x)} + r(x),$1 are determined via $d(x) = d_{p(x)} + r(x),$2-nearest-neighbor PCA in point cloud space.
  • The residual $d(x) = d_{p(x)} + r(x),$3 for the offset $d(x) = d_{p(x)} + r(x),$4 is solved from the planar equation:

  $d(x) = d_{p(x)} + r(x),$5

  with $d(x) = d_{p(x)} + r(x),$6 as camera focal length and principal point, and $d(x) = d_{p(x)} + r(x),$7 as normal components (Zhao et al., 2021).

This approach is parameter-free, runs in real time on CPU (approx. 60 ms/frame), and provides theoretically interpretable predictions. However, it is limited when the local planar assumption fails, sparse geometry is insufficient, or high-frequency texture cues are present in RGB.

4. Outlier Removal and Robustness Procedures

Both learning-based and geometric PR methods benefit from explicit treatment of outlier sparse measurements. The geometric model implements a parameter-free filter designed around LiDAR sampling structure:

• Occlusion Outlier Rejection: Computes expected sensor line spacing, then flags points with depth discontinuities between neighboring scan lines that exceed the plausible angular threshold ($d(x) = d_{p(x)} + r(x),$8). This targets "bleed" where background LiDAR points are incorrectly registered onto foreground image regions.
• Connected Component & Median Cleanup: Post-processes to remove isolated or spurious support points.

This preprocessing consistently reduces RMSE and MAE metrics in ablation on the KITTI benchmark, substantially outperforming prior non-learning depth completion methods (Zhao et al., 2021).

5. Empirical Results and Benchmark Performance

Results on major depth completion datasets establish the efficacy of PR models:

Dataset	Method	RMSE	MAE	iMAE	δ₁
KITTI	PR-Net (Lee et al., 2021)	727.7 mm	186.5 mm	1.72 mm	-
KITTI	Analytic plane-residual (Zhao et al., 2021) †	906.2 mm	257.0 mm	1.09	-
NYU-v2	PR-Net (Lee et al., 2021)	0.104 m	-	-	0.953
NYU-v2	Decoder P only (categorical loss, no R)	0.125 m	-	-	-

† = no RGB used.

Ablation studies demonstrate the critical role of the number of planes $d(x) = d_{p(x)} + r(x),$9 (with $d_{p(x)}$0–$d_{p(x)}$1 optimal) and confirm that residual regression reduces RMSE by over 15% compared to plane-classification-only variants (Lee et al., 2021).

6. Strengths, Limitations, and Extensibility

Plane-residual approaches offer distinctive advantages:

• Efficiency: Single forward pass, compact architecture (4.2M parameters PR-Net vs 38M for CSPN++), and real-time inference (up to 83 fps).
• Interpretability: Division of geometry into coarse–quantized and fine–detailed components.
• Broad Applicability: Direct adaptation to monocular and stereo estimation by reinterpreting planes as depth or disparity bins.

Limitations include the rigidity of global anchor planes for scenes with skewed or broad depth distributions; geometric methods may blur strongly non-planar or textured surfaces. Extensions under exploration include scene-adaptive learned plane placement, RGB-guided superpixel fusion, multi-frame LiDAR fusion, and hybrid models integrating both local analytic planes and learned refinement networks (Lee et al., 2021, Zhao et al., 2021).

7. Context within Depth Completion Methodologies

Plane-residual depth completion unifies concepts from quantized depth binning, local geometric reasoning, and residual regression under a divide-and-conquer framework. Relative to end-to-end regression, the PR paradigm improves stability and boundedness; compared to pure classification, it achieves finer granularity with comparable computational cost. The interpretability, efficiency, and strong empirical results on established benchmarks underscore its value as a foundational approach in practical depth completion applications (Lee et al., 2021, Zhao et al., 2021).