FoundationStereo: Zero-Shot Stereo Depth Model

• FoundationStereo is a large-scale deep stereo-matching model that leverages hybrid features from synthetic data and pretrained monocular depth priors for accurate zero-shot depth estimation.
• It employs a hybrid backbone with transformer and CNN components, using an Attentive Hybrid Cost Filtering module to refine cost volumes and improve disparity predictions through iterative GRU updates.
• The model acts as a teacher for efficient knowledge transfer, enabling dataset distillation and real-time deployment via model compression, NAS-driven acceleration, and structured pruning.

FoundationStereo is a large-scale deep stereo-matching foundation model designed for accurate and robust zero-shot stereo depth estimation. Distinct from conventional stereo approaches, FoundationStereo leverages hybrid feature backbones incorporating both synthetic and foundation-model monocular depth priors, advanced cost volume filtering, and massive, self-curated synthetic datasets. The model defines a new state-of-the-art in zero-shot cross-domain generalization and serves as a "teacher" for efficient knowledge transfer pipelines, dataset curation, and real-time deployment accelerations.

1. Synthetic Dataset Generation and Self-curation

FoundationStereo's preeminence is underpinned by a synthetic dataset comprising 1 million stereo pairs at 1280×720 resolution, generated in NVIDIA Omniverse with path tracing (32–128 spp) and substantial visual diversity—over 5,000 object assets, 12 digital twin scenes, randomized skyboxes, materials, textures, and camera parameters. The data spans both "chaotic" (flying objects, random layouts) and physically plausible (semantic object drops, structured scenes) scenarios. The pipeline includes automated self-curation to eliminate degenerate or ambiguous samples: an initial model is trained on the unfiltered data, each synthetic pair is scored for "bad pixels" (BP-2: error >2 px), and samples with BP-2 >60% are replaced and retrained in two subsequent iterations. This process removes ≈12% of the worst cases, improving generalization metrics (e.g., BP-2 on Middlebury validation set improves from 1.27% to 1.15%) (Wen et al., 17 Jan 2025).

2. Hybrid Network Architecture and Feature Integration

FoundationStereo employs a hybrid backbone that side-tunes a large frozen monocular depth vision transformer (DepthAnythingV2) with a stereo-adaptive CNN (EdgeNeXt-S). This backbone constructs a feature pyramid for both left and right stereo inputs, with fusion at each spatial scale. The resulting feature maps serve as input for a hybrid cost volume, composed of group-wise correlations and concatenated features, forming a 4D representation used for stereo correspondence (Wen et al., 17 Jan 2025, Wen et al., 11 Dec 2025).

A key architectural innovation is the Attentive Hybrid Cost Filtering (AHCF) module. AHCF processes the hybrid cost volume in dual parallel streams: (1) Axial-Planar Convolutions (APC) for local spatial and disparity context, and (2) a Disparity Transformer (DT) applying transformer layers with self-attention across the disparity dimension for long-range reasoning. The outputs from these branches are fused, yielding refined cost volumes and initial disparity estimates via soft-argmin. Subsequent iterative GRU-driven refinements further polish disparity predictions (Wen et al., 17 Jan 2025).

3. Training Objectives and Loss Composition

Training is fully supervised and leverages both synthetic and, via knowledge distillation, unlabeled in-the-wild stereo pairs. The loss function is a weighted sum:

• Photometric loss ($\mathcal{L}_p$), measured using robust penalties (e.g., Charbonnier) on the synthesis error between warped stereo pairs,
• Smoothness regularization ($\mathcal{L}_s$), enforcing piecewise smooth disparities modulated by image gradients,
• Left-right consistency ($\mathcal{L}_c$), penalizing inconsistencies between forward and backward disparity predictions,
• Optionally, direct ground-truth disparity supervision ($\mathcal{L}_d$) (Slezak et al., 5 Jun 2025).

The total loss is $$\mathcal{L}_{FS} = \lambda_p\mathcal{L}_p + \lambda_s\mathcal{L}_s + \lambda_c\mathcal{L}_c + \lambda_d\mathcal{L}_d$$.

Iterative refinement stages are trained with both initial (smooth $L_1$) and refinement ($L_1$) losses, exponentially decayed across steps, enforcing accurate convergence at every update (Wen et al., 17 Jan 2025, Wen et al., 11 Dec 2025).

4. Zero-Shot Generalization and Evaluation

FoundationStereo exhibits high zero-shot generalization: its weights are frozen after pretraining, and no fine-tuning occurs on target domains. Benchmarks on KITTI-12, KITTI-15, Middlebury, and ETH3D demonstrate robust performance with BP-2 and D1 error rates that match or surpass methods fine-tuned on domain-specific data. For instance, on Middlebury and ETH3D, FoundationStereo and its student variants (e.g., "3DGS+FS" distilled pipeline) outperform NeRF-Stereo and raw mesh-derived baselines, especially in the presence of real-world artifacts and challenging context shifts (Slezak et al., 5 Jun 2025, Wen et al., 17 Jan 2025).

Model	KITTI-15 D1 (%)	Middlebury-T Q BP-2 (%)	ETH3D BP-1 (%)
RAFT-Stereo (SceneFlow)	5.46	10.52	2.61
NeRF-Stereo (published)	5.41	8.05	2.94
3DGS (raw)	5.77	10.41	4.65
3DGS+FS (FoundationStereo)	5.52	9.00	2.35

The 3DGS+FS student model strictly outperforms mesh and NeRF-based benchmarks across most settings (Slezak et al., 5 Jun 2025).

5. Expert Knowledge Transfer and Dataset Distillation

FoundationStereo acts as a strong "teacher" for knowledge transfer and synthetic dataset distillation. In the 3D Gaussian Splatting (3DGS) pipeline, clean pseudo-disparity maps are generated by running FoundationStereo inference on synthetic stereo pairs rendered with diverse baselines and focal lengths. These disparity maps serve as "ground-truth" to fine-tune lightweight student stereo networks (e.g., RAFT-Stereo). The process is a pure distillation—no gradient flows to FoundationStereo itself. Using pseudo-labels generated from FoundationStereo, rather than directly from raw 3DGS geometry or reconstructed meshes, yields student models with improved generalization, reducing D1 errors by ~1–1.5% and eliminating artifacts from mesh reconstruction noise (Slezak et al., 5 Jun 2025).

6. Efficiency, Real-Time Adaptations, and Broader Impact

Although FoundationStereo achieves strong accuracy, the baseline model is computationally intensive, running at approximately 0.2 seconds per stereo pair. Fast-FoundationStereo implements a divide-and-conquer acceleration strategy:

• Knowledge distillation compresses the hybrid backbone into a single efficient CNN-based student,
• Blockwise neural architecture search (NAS) automatically discovers latency-optimal cost filtering modules,
• Structured pruning removes redundant channels from the ConvGRU refinement. Automatic pseudo-labeling further supplements training with 1.4M in-the-wild stereo pairs. These optimizations yield real-time inference ($\approx$49 ms per pair) without significant loss in zero-shot performance, surpassing previous real-time designs and matching the original model within a 1% error margin (Wen et al., 11 Dec 2025).

7. Applications and Extensions

FoundationStereo's hybrid depth priors and robust cross-domain generalization motivate its integration into larger multimodal systems. In vision-language-action (VLA) models such as StereoVLA, FoundationStereo's frozen cost volumes serve as geometric input fused with monocular semantic features for downstream manipulation and control tasks, yielding substantial improvements under camera perturbations and complex 3D scenes (Deng et al., 26 Dec 2025). The paradigm extends naturally to omnidirectional stereo, as in DFI-OmniStereo, where foundation-model features regularize cost volume construction for 360° perception, reducing MAE by 16% on real-world panoramic datasets (Endres et al., 30 Mar 2025).

A broader implication is the emergence of a foundation-model paradigm for geometric vision: fusing large-scale, pretrained monocular features via adapters or hybrid convolutional modules into conventional stereo matching pipelines enables both interpretability and state-of-the-art generalization. Such systems can be readily adapted to multi-modal fusion, dataset generation, real-time robotics, and future 3D vision tasks (Slezak et al., 5 Jun 2025, Wen et al., 17 Jan 2025, Deng et al., 26 Dec 2025, Wen et al., 11 Dec 2025, Endres et al., 30 Mar 2025).