Geometric Reciprocity: Unlocking Self-Supervision for Stereoscopic Video Generation

Abstract: Monocular-to-stereo conversion synthesizes stereoscopic content from 2D videos for immersive 3D experiences. In modern Depth-Image-Based Rendering (DIBR) approaches, stereo inpainting of disocclusions is the critical bottleneck. Training-based methods achieve superior quality but rely on scarce stereo pairs or synthetic data with domain gaps. We address this through the first self-supervised framework learning from monocular videos via cycle consistency. Our key contribution is the Geometric Reciprocity Theorem (GRT): under the nearest-neighbor DIBR formulation, the disocclusion mask when synthesizing a target view equals the mask of pixels lost when warping back from target to source, enabling analytical computation of test-time disocclusion masks directly from monocular images. This yields train-test consistency for the stated warping formulation, supporting self-supervised learning from unlimited monocular videos and substantial improvements over training-free and supervised state-of-the-art methods. Project page: https://visual-ai.github.io/grt/

• The paper introduces a novel geometric reciprocity theorem that enables self-supervised stereo inpainting from monocular videos.
• The paper demonstrates that analytic computation of disocclusion masks from depth estimates eliminates the need for error-prone stereo pair data.
• The paper achieves superior performance (PSNR/SSIM/LPIPS of 35.52/0.98/0.0129) and scalability, advancing immersive 3D content generation.

Geometric Reciprocity: Self-Supervised Stereo Inpainting from Monocular Video

Introduction and Motivation

Stereoscopic video generation—especially monocular-to-stereo conversion—remains essential for the proliferation of immersive 3D experiences across VR/AR, cinema, and consumer displays. State-of-the-art methods universally rely on the Depth-Image-Based Rendering (DIBR) pipeline: estimate depth from a monocular frame, warp the source view to the target perspective, and inpaint disocclusions that emerge due to occluded scene content. The inpainting stage, operating on highly structured, geometrically-induced masks, is the limiting factor in output realism and geometric consistency.

All prior training-based progress hits the bottleneck of stereo inpainting data: annotated ground truth requires stereo pair acquisition and accurate per-pixel disocclusion masks. Existing strategies either extract masks via stereo matching—which is error-prone in occluded regions—or synthesize large-scale data by rendering artificial scenes, incurring insurmountable domain shifts. To circumvent this, the authors introduce a fundamentally different approach: self-supervised learning from only monocular videos, enabled by a novel geometric result.

Cycle Consistency and Geometric Reciprocity

The formal core of the paper is the Geometric Reciprocity Theorem (GRT). The authors initially consider cycle consistency as a framework: given a view (e.g., the right-eye image), synthesize the left-eye image, then reconstruct the right-eye image from this pseudo-left via the same DIBR and inpainting pipeline. Cycle consistency loss penalizes deviations between the original and reconstructed target view, serving as a self-supervision signal.

Figure 1: Cycle consistency supervision for monocular-to-stereo: estimate depth from right view $I_R$, synthesize the left via DIBR and inpainting, then complete the cycle by reconstructing the right. The reconstruction loss provides a self-supervised training signal.

Despite its theoretical soundness, direct cycle consistency is computationally prohibitive: multiple network passes, non-differentiable warping, and accumulating errors across the cycle defy efficient large-scale training. The authors uncover that, under nearest-neighbor DIBR, the disocclusion mask required to synthesize a target view (e.g., right-eye from left-eye) equals the mask of pixels lost when warping back from target to source. This insight enables computing training-time inpainting masks purely analytically from a single monocular image and its estimated disparity—eliminating any need for stereo pairs, synthetic data, or the actual cycle.

Figure 2: Progressive reduction of cycle consistency to geometric reciprocity: (i) Left inpainting has no effect on cycle loss; (ii) transferred disparity suffices, eliminating redundancy; (iii) warping is not required, enabling direct mask computation from target-view geometry.

Theoretical Analysis: Geometric Reciprocity Theorem

The GRT is mathematically formalized as:

$$M_{\text{dis}}^{L \to R} = M_{\text{lost}}^{R \to L}$$

where $M_{\text{dis}}^{L \to R}$ is the disocclusion mask for synthesizing the right view from the left, and $M_{\text{lost}}^{R \to L}$ is the set of pixels in the right view that would be lost (due to boundary or occlusion) when warped to the left view. The derivation exploits rectified stereo geometry, discrete warping, and the structure of depth-induced occlusions. Key geometric simplifications reveal that:

• Left inpainting content is irrelevant to the mask.
• Disparity for the left view can be co-transferred from the right, removing the need for a second estimation.
• The warping process can be replaced by direct analytic computation of lost pixels from the target view's disparity.

Consequently, any monocular image can be treated as a 'target' view for self-supervised training: estimate depth, compute the analytically-determined disocclusion mask, and treat the original image content as ground truth for filling those disocclusions.

Figure 3: GRT enables equivalence to full supervision—masks derived analytically from monocular frames yield disocclusion patterns identical to those from actual paired stereo generation, preserving train-inference consistency.

Practical Pipeline and Dataset Construction

The approach results in a scalable and high-fidelity dataset construction paradigm. For any collection of monocular imagery (ImageNet-GRT, Kinetics-GRT, DAVIS-GRT), the process is:

• Treat each frame as a target right-eye view.
• Estimate monocular depth/disparity.
• Compute GRT-predicted disocclusion masks analytically.
• Mask the frame accordingly, and train the inpainting model to reconstruct the original region content.

The authors use state-of-the-art monocular depth estimators to ensure robustness, and the mask computation is independent of downstream stereo inpainting architectures.

Empirical Results

Extensive experiments demonstrate substantial improvements over both supervision-based and training-free baselines. On DAVIS-GRT, the image model achieves PSNR/SSIM/LPIPS of 35.52/0.98/0.0129, exceeding prior state-of-the-art by large margins while being up to two orders of magnitude faster in inference.

Qualitative results reveal that GRT-trained models produce naturalistic textures and preserve fine boundary detail in disoccluded regions—outperforming general inpainting models and previous stereo-specific methods in geometric accuracy and visual plausibility.

Figure 4: Qualitative comparison: GRT-trained inpainting yields superior texture continuity and boundary smoothness versus baselines.

Ablative analyses show the broad applicability of GRT-derived data for various inpainting architectures (LaMa, Stable Diffusion variants, ProPainter), and the performance benefits accrue consistently regardless of the underlying model. Scaling experiments indicate further improvements as more monocular training data is leveraged.

Figure 5: General inpainting models perform well on large masks but consistently fail on the thin, structured disocclusion masks; stereo inpainting via GRT overcomes these limitations.

Figure 6: Stereo matching-based mask extraction yields erroneous disocclusion masks, while GRT provides geometrically accurate and consistent masks for training and inference alignment.

Limitations and Theoretical Implications

The GRT-based approach inherits the limitations of monocular depth estimation and the assumptions of DIBR: inaccuracies in estimated depth, non-Lambertian surfaces, or unrectified stereo geometry may introduce subtle mask errors. The analytic mask computation is exact for nearest-neighbor warping; soft interpolation introduces minor inconsistencies. Importantly, GRT provides not simply a weak self-supervised constraint but a mathematically proven equivalence to full-paired supervision under DIBR, aligning the training and inference domains and eliminating train-test inconsistency.

Theoretically, this work highlights the under-explored power of geometric priors and analytic self-supervision in vision, in contrast to heavy reliance on curated annotated data or synthetic domains. The analytic reduction of the cycle, leading to direct per-image supervision signals, provides a template for other domains with similar geometric symmetry.

Conclusion

This paper advances monocular-to-stereo video generation by introducing the Geometric Reciprocity Theorem, which enables efficient, self-supervised stereo inpainting network training from monocular data alone. The analytic correspondence between disocclusion mask prediction and pixel loss under geometric warping removes the need for stereo pairs or synthetic data, unlocking scalable, high-quality data construction. State-of-the-art empirical results and model-agnostic improvements across architectures substantiate the practical and theoretical utility of the approach.

Future work may extend this analytic geometric supervision to domains beyond stereo, address error modes from monocular depth failure, and enable fully unpaired and scalable 3D content generation for novel-view synthesis and beyond.