Fast FoundationStereo: Real-Time Depth Estimation

• Fast FoundationStereo is a family of stereo matching architectures that combine knowledge distillation, blockwise NAS, and structured pruning to achieve real-time zero-shot generalization.
• It uses an efficient student backbone trained via feature and block-level distillation along with iterative refinement pruning to significantly cut down runtime.
• The approach delivers over 10× speedup compared to heavyweight methods while maintaining competitive performance on benchmarks like Middlebury-Q and KITTI 2015.

Fast FoundationStereo refers to a family of stereo matching architectures and acceleration techniques that achieve strong zero-shot generalization—matching the accuracy of foundation-model-based stereo methods—while operating at real-time frame rates (20+ FPS). This is attained via a combination of knowledge distillation, neural architecture search (NAS) for cost-volume filtering, structured pruning of iterative refinement modules, and training on a large corpus of automatically pseudo-labeled in-the-wild stereo pairs. The approach bridges the gap between heavy, robust foundation stereo models and lightweight—but less generalizable—real-time stereo networks, making robust and accurate stereo depth estimation viable for time-critical applications (Wen et al., 11 Dec 2025).

1. Motivation and Problem Definition

Stereo foundation models such as FoundationStereo deliver high zero-shot generalization across domains by leveraging large-scale synthetic datasets, vision transformer backbones, and explicit injection of monocular priors. However, these models typically exhibit high latency (e.g., 0.7–0.8 s per image pair on an A100 for FoundationStereo’s 32-iteration ConvGRU pipeline), rendering them unsuitable for real-time deployment. Conversely, real-time stereo methods attain fast inference but with a marked drop in robustness and generalization, especially under domain shift, and often require target-domain fine-tuning (Wen et al., 11 Dec 2025, Wen et al., 17 Jan 2025).

Fast FoundationStereo was conceived to provide, for the first time, foundation-model-level generalization and robustness at real-time speeds (>20 FPS), outperforming all previous real-time methods and many heavyweight architectures on a range of benchmarks (Wen et al., 11 Dec 2025).

2. Core Acceleration Techniques

Fast FoundationStereo employs a divide-and-conquer strategy centered around three technical innovations: feature backbone distillation, blockwise neural architecture search, and structured pruning of iterative refinement.

2.1 Knowledge Distillation

Instead of the large, dual-backbone hybrid teacher (e.g., FoundationStereo: DepthAnything ViT with a stereo-adaptive side-tuning CNN), Fast FoundationStereo trains a single highly efficient student backbone (e.g., EdgeNeXt or MobileNetV2 variant) to mimic the key intermediate feature representations of the teacher’s multi-scale pyramid. This is achieved via an MSE-based distillation loss: $$L_\text{feat} = \sum_{i \in \{4,8,16,32\}} \| f_s^{(i)} - \overline{f}^{(i)} \|_2^2$$ where $f_s^{(i)}$ is the student’s feature at scale $i$, and $\overline{f}^{(i)}$ is the teacher’s feature (Wen et al., 11 Dec 2025).

2.2 Blockwise Neural Architecture Search (NAS) for Cost Filtering

The cost filtering network of the teacher (composed of hourglass architectures, axial-planar convolutions, and Disparity Transformers) is decomposed into $N$ sequential blocks. For each block, a candidate set of operations (various 3D convs, residuals, planars, MHA blocks, etc.) is defined. Each candidate is independently distilled to match its teacher block (blockwise MSE loss). NAS is performed via integer linear programming under a total latency budget: $\min \sum_{i=1}^N (\Delta m_i)^\top e_i$ subject to

$$\sum_{i=1}^N (\Delta t_i)^\top e_i \leq \Delta \tau, \quad e_i \in \{\text{one-hot over } C_i\}$$

This reduces search complexity from exponential to linear in the number of block candidates, enabling efficient discovery of optimal architectures under runtime constraints (Wen et al., 11 Dec 2025).

2.3 Structured Pruning of ConvGRU Refinement

The ConvGRU-based iterative disparity refinement module is pruned via global channel pruning. Inter-channel dependencies (e.g., hidden state sharing) are preserved, and importance scores per channel are computed using first-order Taylor expansions. The lowest-importance fraction $\alpha$ of channels is pruned, and the refinement block is retrained with a convex combination of output and intermediate distillation losses (with teacher features frozen). This step significantly reduces the computational cost of refinement while preserving much of the original accuracy (Wen et al., 11 Dec 2025).

3. Large-Scale Pseudo-Labeling and Training Data

A pseudo-labeling pipeline is introduced to supplement synthetic data and further enhance the robustness of the student. Approximately 1.4 million in-the-wild rectified stereo pairs are extracted from large-scale Internet videos (Stereo4D), with frame-striding to maximize diversity. Each pair is processed as follows:

1. Obtain teacher stereo disparity $d_s$ for the left image.
2. Estimate monocular depth $D_m$ via UniDepthV2.
3. Unproject to surface normals $f_s^{(i)}$0 (stereo) and $f_s^{(i)}$1 (mono) using camera intrinsics and Sobel gradients.
4. Compute cosine similarity $f_s^{(i)}$2; threshold to select high-confidence mask $f_s^{(i)}$3 for supervision (e.g., $f_s^{(i)}$4).
5. Set sky pixels to zero disparity using an open-vocabulary segmentation model.

The student is then trained end-to-end on a mix of synthetic datasets, standard benchmarks, and these pseudo-labeled pairs via output-space distillation, which complements feature-level and blockwise architecture distillation (Wen et al., 11 Dec 2025).

4. Model Architecture and Runtime Analysis

The distilled student network comprises:

• A lightweight, single-backbone feature extractor.
• A NAS-discovered cost filtering pipeline (far shallower and more efficient than the teacher’s).
• A globally pruned ConvGRU refinement head.

The total runtime on an NVIDIA-3090 for a Middlebury-Q sized pair is reduced from 496 ms (FoundationStereo) to 49 ms (Fast-FoundationStereo), more than a 10× speedup. Component-wise breakdown is:

• Feature extraction: 243 ms → 20 ms
• Cost filtering: 137 ms → 29 ms
• Refinement (8 steps): 116 ms → 0.4 ms (Wen et al., 11 Dec 2025)

Scalability is maintained: the model fits within 10 GB GPU memory at resolutions up to 4K, with inference remaining sub-second.

5. Quantitative Benchmarking and Generalization

Fast FoundationStereo maintains a narrow performance gap to FoundationStereo, while vastly outperforming previous real-time methods in zero-shot generalization:

Method	Middlebury-Q BP-1%	ETH3D BP-1%	KITTI 2015 D1%	Runtime
FoundationStereo	2.64	0.50	5.20	496 ms
MonSter	7.08	0.99	6.44	336 ms
RT-IGEV	11.28	5.05	7.32	45 ms
Fast-FoundationStereo	4.51	1.22	5.78	49 ms

On non-Lambertian Booster-Q, Fast-FoundationStereo achieves EPE = 1.54 px and BP-4 = 4.62%, almost matching FoundationStereo (4.07%) but at over 10× higher speed. Backbone and cost-filtering ablations demonstrate the effectiveness of distillation and NAS (e.g., M-H BP-2% drops from 2.87% [no distillation] to 2.20% [MSE loss]). Structured pruning trades <0.5 ms of latency for <1% accuracy loss, which is largely recoverable via retraining (Wen et al., 11 Dec 2025).

6. Related Methodologies and Broader Context

Fast FoundationStereo is directly descended from FoundationStereo, which itself pioneered the integration of monocular foundation models as priors in stereo matching, via the side-tuning adapter architecture and hybrid cost-volume plus iterative GRU refinement pipeline (Wen et al., 17 Jan 2025). The entire approach builds upon, and generalizes, principles emerging from fast multi-scale stereo architectures such as FRSNet (Raza et al., 2021), and ViT-adapted methods such as ViTAStereo (Liu et al., 2024), but uniquely combines backbone distillation, blockwise NAS, and structured iterative pruning into a unified framework.

In robotics and embodied AI, these advances have immediate impact: recent work demonstrates that mid-level stereo networks trained via Fast FoundationStereo unlock efficient closed-loop navigation systems with data efficiency far exceeding that of monocular-only NFMs, and dramatically reduce 3D perception ambiguity in urban scenes (Zhou et al., 11 Dec 2025).

7. Future Directions and Limitations

Ongoing research is focusing on quantization (e.g., INT8 or even INT4) to further reduce inference latency and model size, and dynamic per-scene budgets for adaptive runtime/accuracy trade-offs. Automatic pseudo-labeling may be extended to multi-view or video stereo contexts. A plausible implication is that more aggressive fusion of monocular priors (potentially using the structuring mechanisms analyzed in (Yao et al., 20 May 2025)) in tandem with the Fast FoundationStereo pipeline can further lift challenging region performance without adding significant computational cost.

Limitations remain: Fast FoundationStereo, while dramatically faster than its teacher, still trails FoundationStereo slightly in zero-shot accuracy. The pseudo-label curation pipeline, although effective, is subject to errors from both stereo and monocular estimation and relies on hand-tuned thresholds for confidence masking. Extreme lighting, reflections, or low-texture artifacts can still degrade pseudo-label quality (Wen et al., 11 Dec 2025, Wen et al., 17 Jan 2025).

---

References

• "Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matching" (Wen et al., 11 Dec 2025)
• "FoundationStereo: Zero-Shot Stereo Matching" (Wen et al., 17 Jan 2025)
• "Multi-scale Iterative Residuals for Fast and Scalable Stereo Matching" (Raza et al., 2021)
• "Empowering Dynamic Urban Navigation with Stereo and Mid-Level Vision" (Zhou et al., 11 Dec 2025)
• "Diving into the Fusion of Monocular Priors for Generalized Stereo Matching" (Yao et al., 20 May 2025)
• "Playing to Vision Foundation Model's Strengths in Stereo Matching" (Liu et al., 2024)