BaCon-Stereo: Occlusion-Aware Stereo Matching

• The paper introduces a multi-baseline teacher-student architecture that resizes disparity maps to supervise occluded regions effectively.
• It combines geometric consistency, an occlusion-aware attention map, and a contrastive loss to improve training over standard photometric methods.
• Empirical results demonstrate superior occlusion handling and robustness across datasets, outperforming other self-supervised stereo methods.

BaCon-Stereo is a self-supervised stereo matching framework that trains a stereo backbone with multi-baseline contrastive learning, using one reference image and two target images with different baselines in a teacher-student configuration. Its central claim is that regions occluded in the student’s target view are often visible in the teacher’s, so the teacher’s disparity—after baseline rescaling—can supervise both ordinary matching regions and student-occluded regions. The framework combines this geometric consistency with an occlusion-aware attention map and is introduced together with the synthetic multi-baseline dataset BaCon-20k (Xu et al., 14 Aug 2025).

1. Problem setting and motivation

Stereo matching estimates per-pixel disparity between a reference image $I_{ref}$ and a target image $I_{tgt}$, with depth recovered by

$z = \frac{B\cdot f}{d},$

where $B$ is the baseline, $f$ the focal length, and $d$ the disparity. In self-supervised stereo matching, training typically relies on photometric consistency: predict disparity, warp one image into the other, and penalize the reconstruction error, usually with a photometric term and an edge-aware smoothness term (Xu et al., 14 Aug 2025).

The standard photometric objective is

$$\mathcal{L}_{p} = \frac{\alpha}{2}(1-\text{SSIM}(I_{ref}, \hat{I}_{ref})) + (1-\alpha)\left\|I_{ref} - \hat{I}_{ref}\right\|_1,$$

with $\alpha=0.85$, and the smoothness regularizer is

$$\mathcal{L}_{s} = |\partial_x d| \, e^{-|\partial_x I_{ref}|} + |\partial_y d| \, e^{-|\partial_y I_{ref}|}.$$

BaCon-Stereo is motivated by the failure of photometric consistency in occluded regions. For pixels that are visible in the reference view but invisible in the target view, there is no valid correspondence, so photometric supervision becomes misleading. Existing strategies that mask occlusions avoid incorrect supervision but also remove most training signal from precisely the hard regions where disparity completion is needed. The framework explicitly targets three limitations: no reliable self-supervision in occlusions, a single-baseline assumption in teacher-student formulations, and uniform supervision strength across occluded and non-occluded regions (Xu et al., 14 Aug 2025).

2. Multi-baseline teacher-student architecture

BaCon-Stereo uses triplet views consisting of one reference image and two different target images. The student receives $(I_{ref}, I_{tgt}^s)$ with baseline $I_{tgt}$0, while the teacher receives $I_{tgt}$1 with baseline $I_{tgt}$2. The teacher and student share the same reference image but differ in target views; if a target lies to the left, images are flipped horizontally. The teacher is a momentum version of the student, updated by exponential moving average, with no gradients through the teacher (Xu et al., 14 Aug 2025).

The geometry is the key design principle. Since both branches observe the same scene from the same reference view, they share the same depth field, but disparity scales with baseline: $I_{tgt}$3 Defining

$I_{tgt}$4

the teacher disparity is rescaled to the student baseline as

$I_{tgt}$5

This construction creates three supervision regimes. In regions non-occluded for both teacher and student, the teacher provides a geometric pseudo-label that complements photometric supervision. In regions non-occluded for the teacher but occluded for the student, the teacher can provide direct supervision for disparity completion after rescaling. In regions occluded for the teacher, teacher supervision is suppressed because it is unreliable. A plausible implication is that the framework converts baseline diversity into an explicit self-supervised mechanism for handling occlusion, rather than treating occlusion merely as a masking problem (Xu et al., 14 Aug 2025).

3. Contrastive objective and occlusion-aware attention

The framework describes its main consistency term as a negative-free contrastive formulation similar to BYOL: the “positive pair” is the student disparity and the rescaled teacher disparity at the same reference pixel under different target views and baselines. The basic consistency loss is

$I_{tgt}$6

Per pixel,

$I_{tgt}$7

There are no explicit negative samples; collapse is prevented by the auxiliary photometric and smoothness losses (Xu et al., 14 Aug 2025).

Occlusion handling is implemented through teacher and student validity masks, $I_{tgt}$8 and $I_{tgt}$9, derived from photometric thresholding and auto-masking. Teacher photometric loss $z = \frac{B\cdot f}{d},$0 is thresholded with

$z = \frac{B\cdot f}{d},$1

so that high photometric loss indicates a candidate occlusion or poor match. Auto-masking further removes pixels where warping does not improve over the identity mapping.

From these masks, BaCon-Stereo defines an occlusion-aware attention map

$z = \frac{B\cdot f}{d},$2

Thus, teacher-invalid pixels are ignored, jointly valid pixels receive standard consistency weight, and teacher-visible but student-occluded pixels receive doubled weight (Xu et al., 14 Aug 2025).

The student photometric loss is masked as

$z = \frac{B\cdot f}{d},$3

and the weighted contrastive term is

$z = \frac{B\cdot f}{d},$4

The final objective is

$z = \frac{B\cdot f}{d},$5

with

$z = \frac{B\cdot f}{d},$6

This loss decomposition gives BaCon-Stereo its characteristic behavior. In student-occluded but teacher-visible regions, photometric supervision is off, smoothness is weak, and the weighted consistency term dominates. In jointly visible regions, photometric, smoothness, and consistency losses cooperate. This suggests that the framework reallocates learning pressure toward occlusion completion without discarding standard self-supervised reconstruction cues (Xu et al., 14 Aug 2025).

4. Dataset, backbone, and training protocol

BaCon-Stereo is evaluated primarily with IGEVStereo as the stereo backbone. The teacher and student share the same architecture, and the teacher parameters are updated by

$z = \frac{B\cdot f}{d},$7

with a BYOL-style cosine-scheduled momentum $z = \frac{B\cdot f}{d},$8. Data augmentation is applied only to the student inputs and includes color jittering and random occlusion, while the teacher receives clean inputs (Xu et al., 14 Aug 2025).

The supporting dataset, BaCon-20k, is generated in CARLA. It contains 20,417 timestamps across 7 different towns, recorded with five horizontally displaced color cameras, baseline 0.5 m between adjacent cameras, focal length 480 px, and resolution $z = \frac{B\cdot f}{d},$9. The scenes include day and night, varying sun direction, and weather conditions including clear, cloudy, rainy, and foggy. This five-camera setup enables the triplet-view training design by providing one reference image and multiple possible target views with different baselines at each timestamp (Xu et al., 14 Aug 2025).

Training starts from IGEVStereo weights pretrained on SceneFlow, followed by BaCon-20k training on 4× NVIDIA RTX 4090 for 200k iterations with Adam, a one-cycle learning-rate schedule with maximum learning rate $B$0, batch size 16, and random crops of size $B$1. For KITTI fine-tuning, the method generates additional extrapolated views using DEFOM-Stereo disparity warping and Stable Diffusion v2 Inpainting to obtain 4 views and form triplets, then fine-tunes for 1 epoch on 14,184 triplets from KITTI 2015 and 2012 training sets (Xu et al., 14 Aug 2025).

5. Empirical performance and robustness

On KITTI 2015, BaCon-IGEV reports

• NOC D1-BG: 2.06,
• NOC D1-ALL: 2.45,
• ALL D1-ALL: 2.65,

and is described as best among the compared self-supervised methods. On KITTI 2012, it reports

• NOC EPE: 0.5,
• NOC $B$2px: 3.11,
• NOC $B$3px: 1.92,
• ALL $B$4px: 2.31,

again outperforming the listed self-supervised baselines (Xu et al., 14 Aug 2025).

The framework also reports strong cross-dataset robustness. In zero-shot generalization across KITTI 2015, KITTI 2012, Middlebury, and ETH3D, BaCon-IGEV reduces outliers in occluded regions on average by $B$5 compared to the IGEVStereo baseline. On KITTI 2015 specifically, occluded-region outliers are reported as 10.37 for BaCon-IGEV versus 14.26 for IGEVStereo. On Middlebury and ETH3D, the method is described as among the top three across metrics and comparable to DEFOMStereo with $B$6 fewer parameters (Xu et al., 14 Aug 2025).

Weather robustness is a notable part of the evaluation. On DrivingStereo, outliers $B$7px are reported for sunny, cloudy, rainy, and foggy conditions as

• BaCon-IGEV: 2.15 / 1.87 / 5.07 / 1.64,
• IGEVStereo: 4.59 / 5.15 / 15.47 / 4.49.

The rainy-case gap is especially large. The accompanying qualitative discussion attributes this to the robustness of teacher-student geometric consistency in scenes with reflections, water-covered roads, and fog, including conditions not seen explicitly during training (Xu et al., 14 Aug 2025).

The framework is also presented as backbone-agnostic. Applied to other pretrained stereo networks, it yields:

• GwcNet: OCC 29.07 $B$8 12.70, NOC 12.17 $B$9 4.46,
• CFNet: OCC 16.42 $f$0 11.88, NOC 5.87 $f$1 4.26,
• RAFTStereo: OCC 12.70 $f$2 11.73, NOC 5.34 $f$3 3.92 (Xu et al., 14 Aug 2025).

6. Ablations, limitations, and research context

The ablation studies isolate the role of each loss term. On KITTI 2015 training data, using only photometric and smoothness losses gives OCC 97.44, NOC 5.51, ALL 7.14, which the paper interprets as catastrophic failure in occlusions. Using only the contrastive term gives OCC 11.22, NOC 5.23, ALL 5.38. Adding the photometric term improves this to OCC 10.62, NOC 4.49, ALL 4.64, while adding only smoothness instead yields OCC 11.23, NOC 5.14, ALL 5.29. The full model with EMA teacher gives the best result, OCC 10.37, NOC 4.05, ALL 4.21; replacing the EMA teacher with a fixed teacher degrades performance to OCC 10.87, NOC 4.98, ALL 5.12 (Xu et al., 14 Aug 2025).

Ablations on masks and attention show a monotonic pattern. Without thresholding, auto-mask, or attention, the model gives OCC 13.67, NOC 4.93, ALL 5.13. Adding threshold only yields OCC 13.49, NOC 4.76, ALL 4.96. Auto-mask only yields OCC 13.27, NOC 4.43, ALL 4.64. Threshold plus auto-mask gives OCC 12.39, NOC 4.06, ALL 4.25. Adding the full occlusion-aware attention map gives OCC 10.37, NOC 4.05, ALL 4.21. This suggests that attention weighting contributes a substantial fraction of the occlusion improvement beyond simple validity filtering (Xu et al., 14 Aug 2025).

Within the broader self-supervised stereo literature, BaCon-Stereo is positioned against methods that primarily rely on photometric consistency and either ignore occluded pixels or suppress them with masks. The framework differs from teacher-student stereo methods in which teacher and student observe the same stereo pair and therefore the same occlusions, and it differs from third-view methods that use additional viewpoints mainly to improve photometric matching rather than to supervise explicit occlusion completion. Its defining novelty is the use of multi-baseline geometry consistency to make teacher-visible, student-occluded regions directly supervisable (Xu et al., 14 Aug 2025).

The paper also identifies several limitations. Training depends on synthetic multi-baseline data from CARLA, so domain gap remains a concern. The framework requires at least three views per scene, which ordinary binocular datasets do not provide; for KITTI, additional views must be generated by disparity warping and diffusion-based inpainting. Training is computationally heavier because teacher and student are both present during optimization. Future directions mentioned or implied include applying the method to real multi-camera rigs, learning the attention mechanism without fixed photometric thresholds, integrating monocular depth priors or foundation models, and improving multi-view generation for real datasets (Xu et al., 14 Aug 2025).