FeatureNet in FoundationStereo: Hybrid Feature Fusion

• The paper introduces FeatureNet, which integrates frozen ViT features with CNN-derived representations to achieve robust zero-shot stereo matching and self-supervised depth estimation.
• The method employs projection and fusion mechanisms to adapt multi-scale features and construct a compact cost volume for precise disparity refinement under challenging conditions.
• Empirical results demonstrate a reduction in BP2 error from 2.21% to 1.97% and improved sim-to-real robustness through distance masking and side-tuning from monocular priors.

FeatureNet in the FoundationStereo pipeline functions as a hybrid feature-extraction and adaptation module that integrates frozen vision transformer (ViT) features with CNN-derived representations, enabling robust zero-shot stereo matching and self-supervised depth estimation, particularly in challenging visibility conditions such as nighttime scenes. By leveraging large vision foundation models—including DINO and DepthAnythingV2—FeatureNet injects rich semantic priors and geometric context into the stereo matching process, while projection and fusion mechanisms ensure that features are compact and well-suited for subsequent cost-volume construction and disparity refinement (Vankadari et al., 2024, Wen et al., 17 Jan 2025).

1. Architectural Foundations and Motivation

In FoundationStereo (Wen et al., 17 Jan 2025), FeatureNet is architected as the combination of a lightweight CNN backbone (EdgeNeXt-S) and a side-tuning adapter (STA) for frozen foundation models. The pipeline processes rectified stereo pairs $(I_l, I_r) \in \mathbb{R}^{H \times W \times 3}$ at four pyramid levels ($i \in \{4, 8, 16, 32\}$). CNN-derived feature maps $$f_l^{(i)}, f_r^{(i)} \in \mathbb{R}^{C_i \times H/i \times W/i}$$ are concatenated with ViT features extracted at the corresponding scales ($i=4$ in particular for STA operation), then projected to a unified dimension via pointwise convolutions.

In self-supervised nighttime stereo depth estimation (Vankadari et al., 2024), FeatureNet employs a frozen DINO ViT-S/8 as backbone. Image patches are embedded via a $8 \times 8$ conv (stride 4) to form overlapping local features. Six transformer blocks process the patch features $$\hat{f} \in \mathbb{R}^{(H/4) \times (W/4) \times 384}$$ to produce "fine-level" $\tilde{f}$; subsequent downsampling and transformer stages yield "coarse-level" $f \in \mathbb{R}^{(H/8) \times (W/8) \times 384}$, collectively encoding local and global context.

A key design principle is the freezing of ViT/foundation model weights to preserve semantically relevant priors while preventing catastrophic forgetting, with ablations indicating that unfreezing degrades performance (BP2 rising from 1.97% to 3.94% on Middlebury) (Wen et al., 17 Jan 2025).

2. Feature Adaptation and Projection

Both CNN and ViT features are adapted into a stereo-matching–compatible space via channel projection and fusion:

• In FoundationStereo, the STA operates as follows:
  1. Resize image to align with ViT patch size (multiple of 14 for DepthAnythingV2).
  2. Forward through frozen ViT head to obtain last-layer feature $$F_{\text{ViT}} \in \mathbb{R}^{D_V \times H/4 \times W/4}$$.
  3. Project $F_{\text{ViT}}$ to match CNN feature channels via $i \in \{4, 8, 16, 32\}$0 stride-4 convolution, yielding $i \in \{4, 8, 16, 32\}$1.
  4. Concatenate $i \in \{4, 8, 16, 32\}$2 and $i \in \{4, 8, 16, 32\}$3, then apply a $i \in \{4, 8, 16, 32\}$4 conv to obtain the hybrid $i \in \{4, 8, 16, 32\}$5.

• In nighttime stereo (Vankadari et al., 2024), a two-stage projection head (two $i \in \{4, 8, 16, 32\}$6 convolutions with ReLU) reduces ViT features from 384 to 128 channels for both fine- and coarse-level representations. This dimensionality reduction both decreases cost volume computation time and "re-bases" transformer features into distributed embeddings suitable for stereo correlation.

3. Feature Matching and Cost Volume Construction

FeatureNet outputs are integrated into a multi-stage stereo matching pipeline:

• In FoundationStereo (Wen et al., 17 Jan 2025), matching employs both "group-wise correlation" and "concatenation" volumes at $i \in \{4, 8, 16, 32\}$7 resolution to construct a 4D cost volume $i \in \{4, 8, 16, 32\}$8. Channel L2 normalization precedes group-wise correlation:

$i \in \{4, 8, 16, 32\}$9

The hybrid cost volume combines correlation and feature concatenation for each disparity hypothesis.

• In nighttime stereo (Vankadari et al., 2024), a lightweight transformer applies cross- and self-attention along the epipolar direction on projected features $$f_l^{(i)}, f_r^{(i)} \in \mathbb{R}^{C_i \times H/i \times W/i}$$0. A dense correlation volume $$f_l^{(i)}, f_r^{(i)} \in \mathbb{R}^{C_i \times H/i \times W/i}$$1 is constructed via normalized dot product, with a softmax along the disparity axis to obtain matching distributions:

$$f_l^{(i)}, f_r^{(i)} \in \mathbb{R}^{C_i \times H/i \times W/i}$$2

The expected disparity is computed using soft argmax, masked for low-quality matches, and globally refined via feature attention.

4. Refinement, Masking, and Regularization

FeatureNet utilizes multi-level features for both global and local refinement:

• Masking: Distance-based masking suppresses unreliable disparity predictions by thresholding the $$f_l^{(i)}, f_r^{(i)} \in \mathbb{R}^{C_i \times H/i \times W/i}$$3 distance to the nearest distinct neighbor in normalized feature space. This is particularly effective at eliminating spurious matches in regions such as low-texture sky or under extreme lighting (Vankadari et al., 2024).
• Refinement: After masking and initial disparity propagation (attention-based), upsampling to $$f_l^{(i)}, f_r^{(i)} \in \mathbb{R}^{C_i \times H/i \times W/i}$$4 scale is performed, followed by a local matching transformer operating on the warped fine-level features ($$f_l^{(i)}, f_r^{(i)} \in \mathbb{R}^{C_i \times H/i \times W/i}$$5) to predict residual disparity corrections. The final map is optionally re-correlated and upsampled to full resolution via a learned RAFT-style upsampling module.
• Regularization: A distance regularizer $$f_l^{(i)}, f_r^{(i)} \in \mathbb{R}^{C_i \times H/i \times W/i}$$6 encourages spread in feature space, penalizing features that are too close to their nearest neighbor, thereby mitigating feature collapse and improving matchability:

$$f_l^{(i)}, f_r^{(i)} \in \mathbb{R}^{C_i \times H/i \times W/i}$$7

Edge-aware smoothness and photometric losses are also included in the total loss.

5. Integration of Monocular Priors and Sim-to-Real Robustness

A core component of FoundationStereo's FeatureNet is explicit side-tuning to import monocular priors from large-scale ViT-based depth prediction networks (DepthAnythingV2). The resultant hybrid features, fused with learned CNN representations, are empirically shown to boost zero-shot generalization across synthetic to real domains:

• Synthetic data, generated with extensive domain randomization (camera, lighting, texture), is auto-curated to exclude ambiguous matches (BP2 > 60%).
• FeatureNet's fusion of frozen ViT and CNN features narrows the sim-to-real gap, as demonstrated by a ≈10% relative improvement in BP2—falling from 2.21% (CNN+STA only) to 1.97% (full model with all architectural enhancements).
• Model variants using DepthAnythingV2-L as the frozen model outperform DINOv2-L, and freezing the transformer backbone is essential for optimal generalization (Wen et al., 17 Jan 2025).

6. Empirical Properties and Ablation Studies

Extensive ablations highlight the impact of FeatureNet components:

Module/Setting	BP2 (%) Middlebury (Lower is Better)
CNN only	2.48
+ Side-Tuning Adapter (STA)	2.21
+ STA + APC	2.16
+ STA + DT	2.05
+ STA + APC + DT (full)	1.97

FeatureNet's addition of distance-based masking and the distance regularizer $$f_l^{(i)}, f_r^{(i)} \in \mathbb{R}^{C_i \times H/i \times W/i}$$8 provides measurable accuracy gains, especially under adverse nighttime and low-light conditions. The multi-scale use of both mid- and late-layer ViT features increases robustness. Using newer DINOv2 (with 14×14 patch size) underperforms relative to ViT-S/8, likely due to decreased spatial granularity necessary for precise stereo correspondence in low-light (Vankadari et al., 2024, Wen et al., 17 Jan 2025).

7. Synthesis and Impact

FeatureNet in FoundationStereo represents an architectural convergence of foundation vision models and efficient stereo pipelines. By combining multi-scale frozen ViT features with learned CNN representations—fused through side-tuning and projection—FeatureNet delivers strong zero-shot and self-supervised stereo depth estimation. Built-in masking, spatial regularization, and tailored cost-volume construction allow the method to function robustly across synthetic and real domains, and particularly under night/evening and other low-visibility scenarios that pose significant challenges for conventional approaches. Its design principles demonstrate the effectiveness of hybrid feature integration and set a precedent for future research at the intersection of foundation models and geometric vision (Vankadari et al., 2024, Wen et al., 17 Jan 2025).