WAFT-Stereo: Warping-Based Stereo Matching

• The paper introduces WAFT-Stereo, eliminating cost volumes by using a high-resolution warping pipeline combined with a hybrid classification-regression approach for accurate disparity estimation.
• It employs a frozen, pre-trained encoder enhanced via LoRA adapters and iterative regression steps to refine disparity maps over 4–5 iterations.
• Benchmark evaluations on ETH3D, KITTI, and Middlebury show state-of-the-art accuracy with significant computational efficiency gains compared to traditional cost volume methods.

WAFT-Stereo is a warping-based architecture for stereo matching that eliminates the use of either full or partial cost volumes—a central component in prior state-of-the-art stereo models. By leveraging high-resolution feature-space warping combined with a hybrid classification-regression approach, WAFT-Stereo achieves strong performance on major benchmarks with significant gains in computational efficiency. The method was introduced by researchers at Princeton and Stanford, and all concrete pipeline, benchmarking, ablation, and training protocol aspects are available in the original source (Wang et al., 25 Mar 2026).

1. Pipeline and Foundational Principles

WAFT-Stereo is a pure warping-alone field transform (WAFT) pipeline designed to produce dense disparity maps from rectified stereo pairs. The architecture encompasses the following high-level workflow:

1. Feature Extraction: Stereo views $I_L$, $I_R$ are processed via a frozen, pre-trained encoder (DepthAnythingV2), with fine-tuning only through LoRA (rank 8) adapters. Features $F_L$ and $F_R$ are upsampled to half resolution using a DPT head.
2. Initial Disparity Estimation: A one-shot classification module predicts a coarse categorical distribution $P$ over $B$ uniformly spaced disparity bins. The per-pixel initial disparity $d_0$ is computed using a soft-argmax over these bins.
3. Iterative Regression: The initial disparity is iteratively refined via $T-1$ regression steps. Each iteration backward-warps $F_R$ according to the current disparity estimate, concatenates it with $F_L$ and a hidden state $I_R$0, and feeds the combined representation to a recurrent updater module (ViT-small, DPT upsampler, 4 high-resolution ResNet blocks). The module regresses a residual $I_R$1 to update the disparity.

A key methodological claim is that cost volume computation is unnecessary: efficient, differentiable feature warping alone, paired with classification and regression, suffices for accurate and real-time stereo estimation.

2. Mathematical Framework and Warping-Alone Transform

Let $I_R$2 denote left and right rectified images. Feature maps $I_R$3 are extracted at $I_R$4 resolution.

• Backward Warping: For each pixel $I_R$5,

$I_R$6

implemented via differentiable bilinear sampling.

• Iterative Update: Starting with $I_R$7 from classification,

$I_R$8

$I_R$9

$F_L$0

where $F_L$1 is the recurrent updater.

• Classification Initialization: For disparity range $F_L$2 and $F_L$3 bins, bin centers $F_L$4. Given ground-truth $F_L$5,

$F_L$6

Cross-entropy loss supervises $F_L$7 and soft-argmax yields $F_L$8.

3. Eliminating the Cost Volume: Efficiency and Operations

Traditional stereo pipelines construct a 3D cost volume $F_L$9, storing inner products across a disparity range. WAFT-Stereo forgoes this, instead only performing a differentiable warp at each iteration.

Comparative pseudocode:

Paradigm	Step 1	Step 2	Step 3
WAFT-Stereo	$F_R$0	$F_R$1	$F_R$2
Partial cost vol.	Gather costs $F_R$3	$F_R$4	$F_R$5

The computational complexity of WAFT warping is $F_R$6 per iteration, independent of disparity range $F_R$7, while cost volume construction scales as $F_R$8 ($F_R$9: search window) or $P$0 for full range, incurring high memory and compute costs.

4. Architectural and Implementation Details

• Encoder: DepthAnythingV2-L (or S/B variants), frozen except LoRA adapters; DPT head for half-resolution upsampling.
• Classification/Updater Core: Shared ViT-small backbone with 8×8 patch tokens ($P$1), DPT upsampler. The updater additionally employs 4 high-resolution ResNet blocks between hidden states for spatial detail recovery.
• Hidden state: $P$2 across iterations.
• Iterations: $P$3 (1 classification + 4 regression), with a real-time variant using $P$4 and a smaller encoder.
• Regression Head: Outputs parameters for a Mixture-of-Laplace (MoL) distribution. The final residual $P$5 is formed by convex upsampling of the MoL mean.
• Profiling at 540p (NVIDIA L40, BF16):

Variant	Latency (ms)	MACs (T)	Params (B)
WAFT-Stereo (DAv2-S, 4 iters)	47	0.56	0.08
WAFT-Stereo (DAv2-L, 5 iters)	106	2.29	0.38
FoundationStereo	708	12.1	–
S2M2-XL	195	6.26	–

5. Training Methodology

Stage 1 (synthetic pretraining): Uses "SynLarge" aggregate dataset (SceneFlow, FallingThings, FSD, TartanAir, Spring, CREStereo, Sintel, Virtual KITTI-2, UnrealStereo4K, WMGStereo, HR-VS; $P$63.3M pairs). Training is performed on random 480p crops, batch size 32, AdamW optimizer, OneCycle schedule (max lr $P$7), for 400k steps. The multi-part loss combines soft-classification ($P$8) and regression ($P$9) with exponential decay weight $B$0.

Stage 2 (real fine-tuning): KITTI splits ($B$1k steps, batch 16, lr $B$2, $B$3 crops) and Middlebury+InStereo2K+Booster (50k steps, batch 16, lr $B$4, $B$5 crops). Data augmentation includes photometric jitter, horizontal flip, spatter, blur, consistent with recent stereo pipelines.

6. Benchmark Performance and Ablative Analysis

Benchmarks (all with DAv2-L, 5 iters unless noted) demonstrate state-of-the-art results and significant efficiency gains.

Dataset	Metric	Result	Notable Comparison
ETH3D (ZS)	BP-0.5-noc $B$6	0.89% (best)	ZS error $B$7\%
KITTI-2012	BP-2-noc $B$8	1.18%	MonSter++: 1.30%
KITTI-2015	D1-noc $B$9	1.21%	MonSter++: 1.29%
Middlebury	RMSE-noc $d_0$0	5.61 px (best)	–

Efficiency:

• DAv2-L: 10 FPS on 540p (L40).
• DAv2-S: 21 FPS on 540p.
• Speedups of $d_0$1 (S2M2-XL) and $d_0$2 (FoundationStereo).

Ablation studies highlight:

• Optimal bin count: $d_0$3 yields BP-1=1.48%, BP-2=4.62%; higher $d_0$4 brings no gain.
• Regression-only fails (66.8% BP-1). Hybrid classification+regression provides best trade-off.
• 4 high-res ResNet blocks recover fine detail (vs. none: BP-1 rises to 1.99%).
• Best backbone: DepthAnythingV2-L.
• Mixture-of-Laplace loss outperforms $d_0$5 loss for regression.
• Inference cost grows linearly in $d_0$6 (iterations), but $d_0$7–5 suffices in contrast to 32 iterations in prior RAFT-style methods.

7. Limitations and Future Directions

WAFT-Stereo incurs quantization limits from the initial classification: extremely large disparities may require finer binning or a coarse-to-fine refinement strategy. As with pure warping approaches, performance can degrade in highly occluded regions; incorporation of learned occlusion masks is proposed for future work. Further research aims to evaluate larger or multi-scale backbones (DINOv3, Pi3), joint training with optical-flow WAFT for stereo+flow unification, and applications to event-based or cross-modal tasks (infrared, depth completion).

Code and pre-trained weights are publicly available: https://github.com/princeton-vl/WAFT-Stereo (Wang et al., 25 Mar 2026).