Guided RAFT-Stereo: Enhancing Stereo Matching

• The paper introduces guided modifications to the RAFT-Stereo framework, leveraging geometry-aware regularization, LiDAR pre-fill, and global attention to overcome occlusions and ambiguities.
• It iteratively refines disparity estimates using enhanced cost volumes that integrate external priors and contextual cues, leading to faster convergence and improved accuracy.
• Empirical results on datasets like KITTI and Scene Flow demonstrate significant performance gains over standard RAFT-Stereo, especially in textureless, occluded, or repetitive regions.

Guided RAFT-Stereo (GRAFT-Stereo) refers to a class of stereo matching architectures that build upon the iterative refinement paradigm of RAFT-Stereo by integrating additional prior knowledge or geometric/contextual guidance into the feature matching and cost aggregation pipeline. These modifications are designed to address the limitations of conventional RAFT-Stereo—particularly in ill-posed regions such as occlusions, textureless surfaces, and reflective or repetitive patterns—by enhancing the quality of the cost representation presented to the recurrent updater. Several research efforts instantiate this principle, each defining a “guided” variant by either regularizing the matching volume using geometry-aware mechanisms, injecting external measurements (such as sparse LiDAR), or augmenting matching with global attention mechanisms.

1. Theoretical Motivation and Foundation

RAFT-Stereo, and by extension “GRAFT-Stereo,” employs a recurrent, iterative inference cycle that relies on cost-volume retrieval and local feature cues to progressively refine dense disparity estimates. However, the basic RAFT-Stereo design—which repeatedly indexes a fixed, unregularized all-pairs correlation volume—struggles in regions where local appearance provides insufficient or ambiguous disparity cues. Key failure cases include:

• Occlusions, where correspondences are intrinsically unobservable.
• Textureless or reflective areas, where local correlation is unreliable.
• Repetitive structures, leading to ambiguity along the epipolar scanline.

The concept of a guided RAFT-Stereo (“GRAFT-Stereo”) addresses this by either replacing, regularizing, or enhancing the correlation volume with geometry- or context-aware priors; by initializing the iterative process with external measurements; or by augmenting the cost volume or features with global (spatial or epipolar) information. The aim is to improve both initialization and the informativeness of recurrent updates, thus accelerating convergence and raising accuracy, especially under challenging conditions (Xu et al., 2023, Yoo et al., 26 Jul 2025, Li et al., 19 Sep 2025).

2. Algorithmic Variants: Representative Architectures

Three main families of GRAFT-Stereo have been proposed in the literature, each with its own definition of “guidance.”

2.1 Geometry-Encoding Volume Guidance (IGEV-Stereo)

IGEV-Stereo implements GRAFT-Stereo by introducing a geometry-aware, feature-guided cost representation. Starting from the RAFT-Stereo baseline, it replaces the raw all-pairs correlation volume (APC) with a Geometry Encoding Volume (GEV), computed by:

$$\mathbf{C}_{corr}(g,d,x,y) = \frac{1}{N_c/N_g} \langle \mathbf{f}_{l,4}^g(x, y), \mathbf{f}_{r,4}^g(x-d, y) \rangle$$

$\mathbf{C}_{G} = \mathbf{R}(\mathbf{C}_{corr})$

where $\mathbf{R}$ is a lightweight 3D U-Net that regularizes the group-wise correlation volume. Further, the GEV is modulated by left-image features via sigmoid excitation:

$$\mathbf{C}_{i}^{'} = \sigma(\mathbf{f}_{l,i}) \odot \mathbf{C}_i$$

To preserve detail, the GEV is combined with the raw APC pyramid, yielding a Combined Geometry Encoding Volume (CGEV). Disparity refinement then proceeds by sampling from CGEV (rather than only APC) using the current estimate, thus providing context-aware updates (Xu et al., 2023).

2.2 Sparse LiDAR Guidance with Depth Pre-Fill (GRAFT-Stereo)

Another variant leverages sparse LiDAR measurements to guide RAFT-Stereo, addressing the ineffectiveness of naïve sparse disparity injection. Two fusion mechanisms are introduced:

• Late Fusion: The sparse LiDAR-derived disparity map is densified with an interpolation-based pre-fill (e.g., IP-Basic), replacing most zero values with spatially-coherent priors, then used as RAFT-Stereo's initialization. This mitigates high-frequency noise and allows cost-volume retrieval to proceed smoothly.
• Early Fusion: LiDAR points are projected and concatenated (as XYZ) with RGB features before feature encoding; a neural depth completion network generates a confidence-filtered, denser pseudo-depth map to provide accurate geometric cues.

The combined pipeline injects LiDAR-derived geometric priors into both initialization and feature encoding, and empirical results show substantial improvements over both vanilla RAFT-Stereo and other LiDAR-guided methods under sparse regimes (Yoo et al., 26 Jul 2025).

2.3 Global Attention-Based Guidance (GREAT-RAFT)

The GREAT framework generalizes GRAFT-Stereo by integrating global context—both spatial and epipolar—into the matching process:

• Spatial Attention (SA): Projects multi-scale features through local and global self-attention (Outlook and Swin-style), propagating geometric structure across the image.
• Matching Attention (MA): Encodes global epipolar matching context via one-dimensional cross-attention along width (scanline direction) on large-scale features, directly improving cost volume construction.
• Volume Attention (VA): Refines the cost volume using cross-attention between spatial features and disparity search spaces (PDCA).

This guided attention is a plug-in to any iterative stereo backbone (e.g., RAFT-Stereo or IGEV-Stereo). When instantiated with RAFT-Stereo, only MA is used due to architectural constraints (Li et al., 19 Sep 2025).

3. Critical Algorithmic Components

Guidance Type	Mechanism	Principal Effect
GEV/CGEV	3D regularized, feature-excited cost volume	Context-aware, geometry-guided cost
LiDAR Pre-fill	Densified/disambiguated sparse disparity/depth	Improved initialization & features
Attention Modules	Global spatial/epipolar/volume context injection	Disambiguation in ill-posed regions

Each variant retains the core iterative, ConvGRU-based update mechanism of RAFT-Stereo but alters the input cost representation, initial disparity, or feature encoding to incorporate explicit or learned guidance.

4. Empirical Results and Performance Analysis

Experiments across the KITTI, Virtual KITTI 2 (VKITTI2), and MS2 datasets consistently demonstrate that guided RAFT-Stereo mechanisms outperform their unmodified counterparts, particularly under sparse-guidance, ill-posed, or cross-domain regimes.

• IGEV-Stereo achieves EPE 0.47 on Scene Flow, and ranks 1st on KITTI 2015 and KITTI 2012 (Reflective), outperforming vanilla RAFT-Stereo (EPE 0.56) and requiring fewer iterations for convergence, underscoring the benefit of geometry-aware guidance (Xu et al., 2023).
• LiDAR-guided GRAFT-Stereo yields Avg. Err. 0.294 and Bad1 3.33 with 300 points on KITTI, outperforming EG-Depth and SDG-Depth by a significant margin under sparse regimes. Ablation shows that pre-filling is essential for both late and early fusion; without it, sparse signals are buried or become noise-like (Yoo et al., 26 Jul 2025).
• GREAT-RAFT reduces Scene Flow All EPE from 0.551 (RAFT-Stereo) to 0.488, and Occ EPE from 1.938 to 1.747, converges faster, and is especially effective in occluded, textureless, and repetitive regions, validating the utility of global attentional guidance (Li et al., 19 Sep 2025).

5. Mathematical Formulation

GRAFT-Stereo variants extend the RAFT-style cost-volume and recurrent update equations. Canonical forms include:

Cost Volume Construction

$$C(h, w, w') = \sum_j X_l(h, w, j) \cdot X_r(h, w', j)$$

or, for geometry-guided variants: $\mathbf{C}_{G} = \mathbf{R}(\mathbf{C}_{corr})$ or, for attention-guided matching: $$\mathbf{C}_{cat}(\cdot, d, x, y) = \langle \mathbf{f}_{l,4}^{mat}(x, y) \mid \mathbf{f}_{r,4}^{mat}(x-d, y) \rangle$$

Iterative Refinement

$D^{(t)} = D^{(t-1)} + \Delta^{(t)}$

with $\Delta^{(t)}$ obtained from ConvGRU processing of retrieved cost-volume features and contextual encodings.

Loss Functions

$$\mathcal{L}_{init} = \mathrm{Smooth}_{L_1}(\mathbf{d}_0 - \mathbf{d}_{gt})$$

$\mathbf{C}_{G} = \mathbf{R}(\mathbf{C}_{corr})$0

Discontinuity Metric

$\mathbf{C}_{G} = \mathbf{R}(\mathbf{C}_{corr})$1

used to quantify the spatial coherence of disparity map initializations.

6. Experimental Protocols and Evaluation

Standard evaluation employs the KITTI Depth Completion dataset (42,949 train, 3,426 validation, $\mathbf{C}_{G} = \mathbf{R}(\mathbf{C}_{corr})$2 resolution), with synthetic or physically realistic sparse LiDAR inputs (e.g., 4, 8, 16 beams or 300, 1k, 3k uniform samples). Quantitative metrics include Bad1, Bad2 (1- or 2-pixel error threshold), average disparity error (Avg. Err.), RMSE and MAE (for depth). Baselines include RAFT-Stereo, EG-Depth, SDG-Depth, and Selective-RAFT (Yoo et al., 26 Jul 2025, Xu et al., 2023, Li et al., 19 Sep 2025).

Ablation studies demonstrate:

• The necessity of pre-fill for sparse LiDAR guidance: without it, the LiDAR signal is suppressed and performance degrades.
• Trade-offs between interpolation-based and neural pre-filling; smoother, even if less accurate, initializations often yield better downstream refinement for late fusion.
• Early fusion demands high-confidence, accurate geometry to avoid noise propagation.

7. Context, Generalization, and Future Directions

The GRAFT-Stereo paradigm is general: guidance can be geometry-based (3D regularization), measurement-driven (external sensors), or via global attention (context modules). All share the commonality of modifying either the input to, or the iterative mechanism within, RAFT-Stereo to confront its reliance on local, ambiguous cues in difficult scenes.

Recent research generalizes GRAFT-Stereo to broader contexts, including multi-view stereo, domain generalization for diverse driving conditions, and integration with attention mechanisms for broader contextual awareness. It remains a topic of interest to optimize the trade-off between guidance complexity, computational overhead, and universality of the plug-in approach for both indoor and outdoor settings (Xu et al., 2023, Yoo et al., 26 Jul 2025, Li et al., 19 Sep 2025).