Surface Orientation Priors (SGM-P)

• Surface Orientation Priors (SGM-P) are an extension of the SGM framework that incorporate local surface slant to overcome the fronto‐parallel smoothness assumption.
• SGM-P derives priors from coarse stereo estimates, Manhattan-world cues, or ground-truth data and integrates them via an offset in the smoothness penalty.
• Empirical evaluations demonstrate that SGM-P significantly reduces disparity errors and improves reconstruction in challenging scenarios with only modest computational and memory costs.

Surface Orientation Priors (SGM-P) are an extension of the standard Semi-Global Matching (SGM) framework for stereo and multi-view depth estimation, designed to incorporate local surface orientation information into the smoothness regularization. By leveraging explicit prior knowledge about local surface slant, SGM-P overcomes the inherent limitation of SGM's fronto-parallel smoothness assumption, enabling improved disparity/depth estimation in regions with weak texture and significant local tilt. Orientation priors in SGM-P can be derived from coarse-scale stereo, Manhattan-world geometric cues, or ground-truth data, and are integrated efficiently via a shift in the penalty structure of the standard SGM recurrence. This method achieves substantial error reductions in challenging scenarios with minimal memory and computational overhead compared to the baseline SGM approach (Scharstein et al., 2017, Ruf et al., 2019).

1. Mathematical Formulation

The classic SGM algorithm formulates disparity assignment as energy minimization over a 2D Markov Random Field:

$$E(D) = \sum_{p\in I} C(p, d_p) + \sum_{(p, q)\in N} V(d_p, d_q)$$

where $C(p, d_p)$ is the matching cost at pixel $p$ for disparity $d_p$, and $V$ is the smoothness penalty over neighboring disparities. SGM employs a first-order difference penalty:

$$V(d, d') = \begin{cases} 0, & d = d' \ P_1, & |d - d'| = 1 \ P_2, & |d - d'| \ge 2 \end{cases}$$

with user-defined penalties $P_1 < P_2$.

SGM-P augments this formulation by incorporating a per-pixel orientation prior $\pi_p$, modifying the energy to:

$$E_P(D) = \sum_{p\in I} C(p, d_p) + \sum_{(p, q)\in N} V(d_p, d_q) + \lambda \sum_{p\in I} P(d_p; \pi_p)$$

In operational terms, SGM-P realizes this extra term by shifting the arguments in $V$:

$$L_r(p, d) = C(p, d) + \min_{d'} [ L_r(p - r, d') + V(d + j_r(p, d), d') ]$$

where $j_r(p, d)$ encodes the local predicted disparity step according to the surface prior. In multi-view SGM-P, the shift is determined from the local surface normal $n_p$ with a discrete offset $\Delta n(p)$, modifying the standard penalty to:

$$P^{(\rm surf)}_{\rm sgm}(d_p, d_q; n_p) = P_{\rm sgm}(d_p + \Delta n(p), d_q)$$

This formulation constrains disparity transitions along scanlines to favor those coherent with the predicted 3D orientation.

2. Representations and Derivation of Orientation Priors

Three principal sources are used to derive surface orientation priors in SGM-P:

• Plane-fitting priors from coarse SGM results: Running SGM at low resolution, the resulting disparity is segmented into planar patches using superpixels or RANSAC. Planes parameterized as $d(u, v) = a u + b v + c$ serve as local priors. Each pixel inherits parameters $(a, b, c)$ from its assigned plane, or multiple planes if ambiguity exists.
• Manhattan-world normal priors: Global scene directions are extracted using vanishing point detection and line clustering, assigning each pixel one of three Manhattan directions. These normals are integrated via least-squares into a depth surface $Z(u, v)$, which is converted into candidate disparity surfaces $d(u, v) = b f / Z(u, v)$.
• Oracle priors from ground-truth data: The true disparity map $D_{\rm gt}(u, v)$ or its piecewise-planar approximation is used directly as a prior for benchmarking upper bound performance and analyzing the bias introduced by imperfect prior estimation (Scharstein et al., 2017).

3. Integration into the SGM Pipeline

SGM-P modifies the scanline dynamic programming of SGM:

• For each scan direction $r$, the offset $j_r(p, d)$ is precomputed from prior $\pi_p$ or normal $n_p$.
• The recurrence for the path cost $L_r(p, d)$ is adjusted so that the local cost for transitioning between disparities is computed not relative to a fronto-parallel model, but relative to the local surface orientation. For 2D priors, the offset depends only on $(u, v)$; for 3D priors, it is disparity-dependent.
• Aggregation over all scan directions yields the total cost $S(p, d)$ for each pixel-disparity pair.

Pseudocode outline:

for each direction r in R: generate offset j_r(p,·) from π_p for each pixel p along a scanline in r: for each disparity d: L_r(p,d) = C(p,d) + min_{d'} [L_r(p–r,d') + V(d + j_r(p,d), d')] for each pixel p: S(p,·)=∑_r L_r(p,·) dₚ = argmin_d S(p,d)

SGM-P thus incorporates prior-induced offsets directly into the cost aggregation, incurring only modest additional memory (for the offset images/volumes) and computational cost.

4. Parameterization and Implementation

Key algorithmic parameters include:

• Disparity range $\mathcal{D}$: Chosen to cover scene depth (e.g., 0–255 pixels).
• Matching cost $C(p, d)$: 5 × 5 normalized cross-correlation (NCC) with truncation and stabilization for textureless regions: $$C(p, d) = 255 \cdot (1-\max(0, \mathrm{NCC}(p, d)))$$.
• Smoothness penalties: $P_1 = 100$; $P_2 = P_1(1 + \alpha \exp(-|\Delta I|/\beta))$, with $\alpha = 8, \beta = 10$, and $\Delta I$ the absolute intensity difference between neighbors.
• Prior weight $\lambda$: Effectively realized by the magnitude of the offset $j_r$, making an explicit global weight unnecessary.
• Memory and runtime costs: Additional storage for offset images (per scan direction in 2D) or offset volumes ($|I| \times |\mathcal{D}|$ for 3D), with measured runtime overhead ~7% for EPi/EPv variants and negligible for ground-truth surface priors.
• Implementation: SGM-P is compatible with either CPU or GPU parallelization, with performance demonstrated at 1–2 Hz for 1920×1080 imagery in a GPU realization (Ruf et al., 2019).

5. Empirical Performance and Comparative Analysis

Experimental evaluations have utilized high-resolution Middlebury benchmark pairs and multi-view datasets for quantitative and qualitative assessment:

• Middlebury high-res (Adirondack, Motorcycle, Playroom, Vintage): SGM-P (SGM-EPi) reduces error rates by 13–41% (e.g., 28.4% → 20.3% on Adirondack) relative to SGM at 100% completeness; oracle priors (SGM-GS) achieve up to 80% reduction (Scharstein et al., 2017).
• Full Middlebury training set: Error reduction for SGM-EPi ranges from –1% to 41%, mean ≈12%, with no severe degradations.
• Manhattan-world priors (SGM-MW): Enables accurate reconstruction of smooth, slanted, untextured surfaces, outperforming standard SGM and lane-based planar priors in scenes exhibiting strong orthogonality.
• Multi-view SGM-P: Surface-aware SGM using joint normal and depth estimation enhances consistency and raises ROC curves, particularly on slanted roofs and facades in aerial imagery (Ruf et al., 2019).
• Cost function agnosticism: SGM-P yields similar gains with advanced matching costs, such as MC-CNN, confirming that benefits stem from improved smoothness modeling rather than the choice of cost volume.
• Online performance: SGM-P supports incremental, online computation—unlike global bundle adjustment approaches (e.g., COLMAP)—with frame rates competitive for aerial image augmentation.

6. Scope, Limitations, and Further Directions

Performance gains from SGM-P concentrate on slanted, weakly-textured surfaces where fronto-parallel regularization fails. The technique is robust on scenes dominated by fronto-parallel structure or high texture, causing neither adverse effects nor substantial improvements. Quality of estimated priors impacts effectiveness; gross misfits in plane segmentation or normal estimation locally diminish accuracy.

2D priors cannot address overlapping planes at depth discontinuities; 3D priors (SGM-EPv, SGM-GNv) better capture abrupt depth transitions but have increased computational requirements. SGM-P is not designed to handle highly curved or non-piecewise-planar surfaces, nor does it incorporate higher-order (second-order) or learned MRF smoothness. Open research questions involve the generation of more robust, semantic, or learned priors, the extension to curved geometries, and unified formulations leveraging both orientation priors and higher-order regularization (Scharstein et al., 2017).

7. Recommendations and Best Practices

For practical deployment, piecewise-planar priors from a coarse SGM pass (SGM-EPi) are recommended in high-resolution images with large homogeneous surfaces. If scene geometry exhibits Manhattan-world structure, integrating those normals (SGM-MW) is beneficial in textureless or planar-degenerate regions. Retaining standard SGM cost and penalty parameters with inserted offset shifts obviates the need for additional hyperparameter tuning.

For future research and application, combining efficient, surface-aware SGM-P with advances in semantic segmentation, deep priors, or higher-order discrete regularizers remains a promising avenue for further reducing error and increasing geometric fidelity in challenging stereo and multi-view scenarios (Scharstein et al., 2017, Ruf et al., 2019).