Stereo Regularization in Depth Estimation

• Stereo regularization is a framework that applies priors, penalties, and learned cost aggregation to convert ambiguous stereo correspondences into coherent depth maps.
• It integrates classical methods like edge-aware smoothness and graph cuts with deep architectures such as 3D CNNs and CRFs for improved disparity estimation.
• Recent approaches leverage architectural regularization and self-supervised losses to enhance accuracy and efficiency on benchmarks like KITTI, Middlebury, and DTU.

Searching arXiv for relevant stereo-regularization papers and IDs. arXiv search: stereo regularization cost volume CRF soft argmin stereo matching Searching arXiv for the cited papers by title to verify IDs and recency. Stereo regularization denotes the family of priors, penalties, architectural mechanisms, and cross-view consistency constraints that transform ambiguous matching evidence into a coherent disparity or depth field. In rectified stereo, disparity $d(x,y)$ is recovered by comparing left and right images along epipolar lines, but the inference problem is ill-posed in textureless areas, repetitive patterns, reflective or transparent regions, and near occlusion boundaries. The literature uses “regularization” in both classical and modern senses: as explicit smoothness or visibility terms in an energy, as learned cost-volume aggregation and iterative refinement, and, in more recent work, as geometry-aware supervision for multi-view reconstruction, neural rendering, 3D Gaussian Splatting, and monocular-to-stereo video generation (Xu et al., 2023, Zhang et al., 2014).

1. Problem definition and the need for regularization

The classical stereo formulation treats disparity estimation as a correspondence problem constrained by epipolar geometry. Reliable matches are sparse in many regions, so local photometric evidence alone is insufficient. A typical historical remedy is edge-aware smoothness, for example

$$L_{\text{smooth}}=\sum_{x,y}\left(|\partial_x d|e^{-|\partial_x I|}+|\partial_y d|e^{-|\partial_y I|}\right),$$

optimized with graph cuts, CRFs, diffusion, or semi-global matching, which aggregates pairwise costs along multiple paths to approximate global regularization (Xu et al., 2023).

Deep stereo preserves the same basic ill-posedness but shifts regularization into learned representations. Instead of hand-crafted matching costs, neural methods construct a cost volume $C(x,y,d)$ and regularize it by learned aggregation, recurrent updates, or structured inference. In the formulation of GC-Net, stereo can still be viewed through the lens of a classical data term plus regularizer, but the regularizer is implicit in 3D convolutions over a deep cost volume rather than an explicit hand-designed prior (Kendall et al., 2017).

A recurring clarification in the literature is that stereo regularization is not synonymous with a smoothness loss. In some systems, it is an explicit penalty; in others, it is the behavior induced by a 3D CNN, a CRF, a recurrent updater, a graph Laplacian, or a cross-view cycle constraint. This distinction is central to understanding why methods with very different objective functions can all be described as regularized stereo estimators.

2. Classical formulations: smoothness, scale coupling, visibility, and discrete optimization

Classical regularization is often presented as local smoothness, but the literature contains broader formulations. “Cross-Scale Cost Aggregation for Stereo Matching” reformulates cost aggregation as a weighted least-squares problem and adds an inter-scale regularizer independent of the similarity kernel, producing a generic cross-scale framework that can be applied to box filtering, bilateral filtering, guided filtering, non-local means, and tree-based aggregation. The resulting coarse-to-fine coupling improved multiple state-of-the-art aggregation methods on Middlebury, KITTI, and New Tsukuba, while adding at most about $14.3\%$ overhead relative to single-scale aggregation (Zhang et al., 2014).

A more radical reformulation appears in “A new stereo formulation not using pixel and disparity models,” where the sites are gaze lines rather than pixels and the labels are depth numbers rather than disparities. In that formulation, visibility reasoning is naturally included in the energy, and a small convex smoothness term is sufficient because the pairwise term inhibits jumps greater than one depth index between neighboring sites. This yields exact graph-cut optimization in the base formulation and very high-speed GPU implementations; for the Tsukuba stereo pair, the reported runtime is $5\text{ ms}$ on GTX1080GPU and $19\text{ ms}$ on GTX660GPU (Oguri et al., 2018).

The optimization view remains active in more recent nonstandard settings. “Quantum-Hybrid Stereo Matching With Nonlinear Regularization and Spatial Pyramids” formulates stereo as MRF/MAP inference with an edge-aware truncated linear regularizer,

$$E_s(d_{i,j}, d_{i',j'})= \begin{cases} R & \text{if } |I^L(i,j)-I^L(i',j')|\le \tau,\ R/q & \text{else}, \end{cases} \qquad R=\min(m,s|d_{i,j}-d_{i',j'}|),$$

maps the problem to QUBO, and solves it in a quantum-classical hybrid pipeline with a spatial pyramid. On Middlebury, the paper reports improved root mean squared accuracy over the previous state of the art in quantum stereo matching of $2\%$ and $22.5\%$ when using different solvers (Braunstein et al., 2023).

Taken together, these works show that classical stereo regularization encompasses at least four distinct ideas: local coherence, cross-scale coupling, explicit visibility structure, and tractable discrete optimization under carefully designed pairwise terms.

3. Learned cost-volume regularization and architecture-level priors

Deep stereo regularization first became prominent through explicit cost-volume filtering. GC-Net constructs a deep cost volume by concatenating rectified unary features across disparities, applies a 3D encoder-decoder over $(x,y,d)$, and regresses disparity with a differentiable soft argmin,

$$L_{\text{smooth}}=\sum_{x,y}\left(|\partial_x d|e^{-|\partial_x I|}+|\partial_y d|e^{-|\partial_y I|}\right),$$0

The paper’s central claim is that geometry is encoded by epipolar alignment in the volume construction, while context is learned by 3D convolutions that aggregate evidence over height, width, and disparity; in this view, learned 3D context replaces hand-engineered SGM, MRF, or CRF regularization (Kendall et al., 2017).

A closely related but more explicitly structured approach appears in “Deep Stereo Matching with Dense CRF Priors.” There, a Siamese CNN produces patch-wise matching costs, and a densely connected CRF acts as a prior on inter-pixel interactions. Fully connected bilateral and spatial Gaussian kernels are embedded into mean-field inference unrolled as a recurrent network, so regularization becomes an end-to-end differentiable layer rather than post-processing. The paper reports that CRF regularization improves edge fidelity and reduces streaking artifacts, and that CRF combined with SGM can improve the $$L_{\text{smooth}}=\sum_{x,y}\left(|\partial_x d|e^{-|\partial_x I|}+|\partial_y d|e^{-|\partial_y I|}\right),$$1-pixel error relative to MC-CNN plus SGM alone (Slossberg et al., 2016).

IGEV-Stereo pushes this trajectory toward hybrid architectural regularization. It combines a lightweight 3D UNet regularizer, producing a Geometry Encoding Volume (GEV), with RAFT-style iterative refinement over a Combined Geometry Encoding Volume that fuses GEV and raw all-pairs correlations. The GEV is also used for soft-argmin initialization,

$$L_{\text{smooth}}=\sum_{x,y}\left(|\partial_x d|e^{-|\partial_x I|}+|\partial_y d|e^{-|\partial_y I|}\right),$$2

which provides a geometry-aware starting point for ConvGRU updates. The method uses supervised losses only and explicitly does not use additional smoothness, edge-aware, or occlusion penalties. Empirically, it reports Scene Flow test EPE $$L_{\text{smooth}}=\sum_{x,y}\left(|\partial_x d|e^{-|\partial_x I|}+|\partial_y d|e^{-|\partial_y I|}\right),$$3, KITTI 2012 runtime $$L_{\text{smooth}}=\sum_{x,y}\left(|\partial_x d|e^{-|\partial_x I|}+|\partial_y d|e^{-|\partial_y I|}\right),$$4 with EPE-all $$L_{\text{smooth}}=\sum_{x,y}\left(|\partial_x d|e^{-|\partial_x I|}+|\partial_y d|e^{-|\partial_y I|}\right),$$5, and KITTI 2015 D1-all $$L_{\text{smooth}}=\sum_{x,y}\left(|\partial_x d|e^{-|\partial_x I|}+|\partial_y d|e^{-|\partial_y I|}\right),$$6 at $$L_{\text{smooth}}=\sum_{x,y}\left(|\partial_x d|e^{-|\partial_x I|}+|\partial_y d|e^{-|\partial_y I|}\right),$$7; it is described as ranking first on KITTI 2015 and 2012 among published methods and as the fastest among the top 10 methods (Xu et al., 2023).

This line of work makes a key conceptual point: in deep stereo, regularization is often architectural. A 3D CNN, a dense CRF, or a geometry-aware recurrent updater can regularize matching even when no explicit smoothness term appears in the loss.

4. Edge-aware, confidence-aware, and refinement-centric regularization

A second major strand centers on explicit refinement mechanisms that preserve discontinuities while removing noise. EdgeStereo combines a disparity branch with an edge sub-network, using a context pyramid, a residual pyramid, feature embedding of edge cues, and an edge-aware smoothness loss. Its residual recursion is

$$L_{\text{smooth}}=\sum_{x,y}\left(|\partial_x d|e^{-|\partial_x I|}+|\partial_y d|e^{-|\partial_y I|}\right),$$8

and the explicit disparity smoothness term is

$$L_{\text{smooth}}=\sum_{x,y}\left(|\partial_x d|e^{-|\partial_x I|}+|\partial_y d|e^{-|\partial_y I|}\right),$$9

where $C(x,y,d)$0 is the learned edge probability map. The paper applies this loss only in Phase 2 with $C(x,y,d)$1 and removes it in Phase 3 to avoid instability during joint training. On Scene Flow, the full model reports EPE $C(x,y,d)$2 and $C(x,y,d)$3 px error $C(x,y,d)$4; on KITTI 2015 it reports D1-all $C(x,y,d)$5 with runtime $C(x,y,d)$6 (Song et al., 2018).

“Learned Collaborative Stereo Refinement” casts disparity denoising as a variational problem in a joint disparity, color, and confidence image space. The energy is

$C(x,y,d)$7

and the resulting variational network is obtained by unrolling proximal-gradient iterations. The regularizer is a multi-scale Fields-of-Experts model over the stacked channels, while the data term contains a confidence-weighted disparity fidelity. Confidence is derived from the probability volume and a left-right consistency check, so refinement is guided by both color structure and estimated reliability. On KITTI 2015, the paper reports bad3 improving from initial WTA values of occ $C(x,y,d)$8, noc $C(x,y,d)$9 to occ $14.3\%$0, noc $14.3\%$1 for a strong variational-network configuration (Knöbelreiter et al., 2019).

These methods correct a common oversimplification: boundary preservation does not require only image-gradient weights. Learned edge probabilities, confidence maps, joint RGB–disparity–confidence filters, and proximal inference can all serve as regularizers, provided they suppress fluctuations in weakly constrained regions without erasing discontinuities.

5. Self-supervision, domain adaptation, and consistency-driven regularization

Stereo regularization is especially visible when supervision is weak or absent. “Semi-supervised learning of deep metrics for stereo reconstruction” uses stereo constraints themselves as regularizers for a deep patch metric. The method enforces epipolar geometry, disparity range, uniqueness, ordering, continuity, and occlusion handling through constrained dynamic programming and hinge-style losses. More constraints systematically improve performance: on KITTI’12 WTA error, MIL gives $14.3\%$2, CONTRASTIVE gives $14.3\%$3, MIL-CONTRASTIVE gives $14.3\%$4, and CONTRASTIVE-DP gives $14.3\%$5; the semi-supervised metric then reaches $14.3\%$6 on KITTI’12 versus $14.3\%$7 for the supervised MC-CNN-fst baseline (Tulyakov et al., 2016).

In unsupervised monocular depth prediction trained from stereo, regularization appears as bilateral cyclic consistency and adaptive weighting. “Bilateral Cyclic Constraint and Adaptive Regularization for Unsupervised Monocular Depth Prediction” uses photometric reconstruction, structural similarity, edge-aware smoothness, and a bilateral cycle loss $14.3\%$8, with adaptive weights

$14.3\%$9

so that regions with large reconstruction residuals, such as occlusions and dis-occlusions, are softly down-weighted rather than explicitly masked. The paper reports that replacing left-right consistency with bilateral cyclic consistency improves performance further, especially D1-all and SqRel (Wong et al., 2019).

Domain adaptation introduces another interpretation of regularization. StereoGAN jointly optimizes domain translation and stereo matching, adding bidirectional multi-scale feature re-projection, correlation consistency, and mode-seeking regularization so that translated stereo pairs remain epipolar-consistent. On Synthia $5\text{ ms}$0 KITTI2015 with a DispNet backbone, the reported D1-all decreases from $5\text{ ms}$1 for direct inference and $5\text{ ms}$2 for CycleGAN adaptation to $5\text{ ms}$3 for StereoGAN (Liu et al., 2020).

ZOLE regularizes self-adaptation by exploiting scale diversity. It observes that up-sampled target-domain stereo pairs often produce disparities with extra details, then uses those outputs as pseudo-labels while imposing a graph Laplacian regularizer that preserves desired edges and smooths artifacts. On a smartphone stereo dataset, the reported PSNR/SSIM rises to $5\text{ ms}$4, compared with $5\text{ ms}$5 for the variant without Laplacian regularization and $5\text{ ms}$6 for DispNetC (Pang et al., 2018).

These works suggest that in low-label or cross-domain regimes, stereo regularization becomes primarily a question of geometry consistency: constraints on ordering, cycles, feature reprojection, and graph structure take over the role traditionally played by hand-designed smoothness terms.

6. Multi-view stereo, neural scene representations, and cross-view generative models

In multi-view stereo, regularization must act along the depth dimension as well as across image space. NR2-Net regularizes MVS cost maps with sliding depth blocks, depth attention within each block, and gated recurrence across blocks, thereby modeling non-local depth interactions without full 3D cost-volume memory cost. It also introduces a dynamic depth-map fusion strategy. On DTU, the reported result is Acc. $5\text{ ms}$7, Comp. $5\text{ ms}$8, Overall $5\text{ ms}$9; on Tanks and Temples Intermediate, the method reports Acc. $19\text{ ms}$0, Comp. $19\text{ ms}$1, and F$19\text{ ms}$2 $19\text{ ms}$3 (Xu et al., 2021).

The same principle extends to neural implicit reconstruction. S-VolSDF regularizes VolSDF with an MVS probability volume and a generalized cross entropy loss,

$19\text{ ms}$4

where $19\text{ ms}$5 is a soft multi-view-consistent MVS probability and $19\text{ ms}$6 is the neural rendering termination weight along the ray. The paper argues that this constrains shape–radiance ambiguity under sparse views and reports DTU Chamfer distance $19\text{ ms}$7 for S-VolSDF with CasMVSNet, compared with $19\text{ ms}$8 for VolSDF and $19\text{ ms}$9 for CasMVSNet alone (Wu et al., 2023).

Stereo regularization also appears in sparse-view 3D Gaussian Splatting. StereoGS constructs virtual stereo pairs during optimization, applies a foundation stereo model, converts disparity to metric depth through

$$E_s(d_{i,j}, d_{i',j'})= \begin{cases} R & \text{if } |I^L(i,j)-I^L(i',j')|\le \tau,\ R/q & \text{else}, \end{cases} \qquad R=\min(m,s|d_{i,j}-d_{i',j'}|),$$0

and supervises rendered inverse depth with a validity mask filtering occlusions, background, and anomalous disparities. It complements this with Gradient-Aware Opacity Decay and a Consistency-Aware Dense Initialization from zero-shot multi-view depth. The paper reports state-of-the-art sparse-view performance on LLFF, DTU, Mip-NeRF360, and Blender, without additional inference overhead (Yuan et al., 29 Jun 2026).

In generative video, StereoWorld treats geometry as an explicit training target rather than as a downstream post-processing signal. Its total loss combines RGB diffusion, depth diffusion, and disparity supervision:

$$E_s(d_{i,j}, d_{i',j'})= \begin{cases} R & \text{if } |I^L(i,j)-I^L(i',j')|\le \tau,\ R/q & \text{else}, \end{cases} \qquad R=\min(m,s|d_{i,j}-d_{i',j'}|),$$1

with $$E_s(d_{i,j}, d_{i',j'})= \begin{cases} R & \text{if } |I^L(i,j)-I^L(i',j')|\le \tau,\ R/q & \text{else}, \end{cases} \qquad R=\min(m,s|d_{i,j}-d_{i',j'}|),$$2, $$E_s(d_{i,j}, d_{i',j'})= \begin{cases} R & \text{if } |I^L(i,j)-I^L(i',j')|\le \tau,\ R/q & \text{else}, \end{cases} \qquad R=\min(m,s|d_{i,j}-d_{i',j'}|),$$3, and $$E_s(d_{i,j}, d_{i',j'})= \begin{cases} R & \text{if } |I^L(i,j)-I^L(i',j')|\le \tau,\ R/q & \text{else}, \end{cases} \qquad R=\min(m,s|d_{i,j}-d_{i',j'}|),$$4. The paper’s ablation shows EPE decreasing from $$E_s(d_{i,j}, d_{i',j'})= \begin{cases} R & \text{if } |I^L(i,j)-I^L(i',j')|\le \tau,\ R/q & \text{else}, \end{cases} \qquad R=\min(m,s|d_{i,j}-d_{i',j'}|),$$5 without depth or disparity regularization to $$E_s(d_{i,j}, d_{i',j'})= \begin{cases} R & \text{if } |I^L(i,j)-I^L(i',j')|\le \tau,\ R/q & \text{else}, \end{cases} \qquad R=\min(m,s|d_{i,j}-d_{i',j'}|),$$6 with both, while D1-all decreases from $$E_s(d_{i,j}, d_{i',j'})= \begin{cases} R & \text{if } |I^L(i,j)-I^L(i',j')|\le \tau,\ R/q & \text{else}, \end{cases} \qquad R=\min(m,s|d_{i,j}-d_{i',j'}|),$$7 to $$E_s(d_{i,j}, d_{i',j'})= \begin{cases} R & \text{if } |I^L(i,j)-I^L(i',j')|\le \tau,\ R/q & \text{else}, \end{cases} \qquad R=\min(m,s|d_{i,j}-d_{i',j'}|),$$8 (Xing et al., 10 Dec 2025).

The term also generalizes beyond photographic stereo. In neural optical flow for stereo PIV, regularization means sharing one continuous world-space velocity field across both cameras, so that the same smoothness, physics constraints, and spectral bias apply to each view. On stereo 3D HIT, the reported NRMSE is $$E_s(d_{i,j}, d_{i',j'})= \begin{cases} R & \text{if } |I^L(i,j)-I^L(i',j')|\le \tau,\ R/q & \text{else}, \end{cases} \qquad R=\min(m,s|d_{i,j}-d_{i',j'}|),$$9 for NOF, versus $2\%$0 for wavelet-based optical flow and $2\%$1 for cross-correlation (Masker et al., 2024).

Across these literatures, stereo regularization has become a general principle for enforcing binocular or multi-view consistency in whatever representation is being optimized: disparity volumes, recurrent depth states, implicit surface termination probabilities, Gaussian primitives, generative video latents, or neural-implicit flow fields. A plausible implication is that the historical opposition between “regularization” and “representation” has largely dissolved: in current systems, the regularizer is often the mechanism that makes the representation geometrically meaningful in the first place.