Continuous Cost Aggregation for Dual-Pixel Disparity Extraction

Abstract: Recent works have shown that depth information can be obtained from Dual-Pixel (DP) sensors. A DP arrangement provides two views in a single shot, thus resembling a stereo image pair with a tiny baseline. However, the different point spread function (PSF) per view, as well as the small disparity range, makes the use of typical stereo matching algorithms problematic. To address the above shortcomings, we propose a Continuous Cost Aggregation (CCA) scheme within a semi-global matching framework that is able to provide accurate continuous disparities from DP images. The proposed algorithm fits parabolas to matching costs and aggregates parabola coefficients along image paths. The aggregation step is performed subject to a quadratic constraint that not only enforces the disparity smoothness but also maintains the quadratic form of the total costs. This gives rise to an inherently efficient disparity propagation scheme with a pixel-wise minimization in closed-form. Furthermore, the continuous form allows for a robust multi-scale aggregation that better compensates for the varying PSF. Experiments on DP data from both DSLR and phone cameras show that the proposed scheme attains state-of-the-art performance in DP disparity estimation.

• The paper introduces Continuous Cost Aggregation, a method that fits parabolic models to matching costs for precise sub-pixel disparity extraction in dual-pixel sensors.
• The approach employs multi-scale disparity fusion with quadratic constraints, effectively compensating for varying point spread functions across resolutions.
• Performance evaluations on DSLR and mobile-phone images demonstrate state-of-the-art accuracy without extensive training, underscoring its real-world applicability.

Continuous Cost Aggregation for Dual-Pixel Disparity Extraction

Introduction

The paper introduces a novel method for extracting disparity information from dual-pixel (DP) sensors, which are increasingly prevalent in modern cameras. Unlike conventional stereo cameras, DP sensors provide two views from a single shot, resulting in a stereo pair with a minimal baseline. This unique characteristic poses challenges for traditional stereo matching algorithms due to the differential point spread function (PSF) per view and the limited disparity range. The proposed method, Continuous Cost Aggregation (CCA), addresses these issues by integrating a continuous cost optimization approach within a semi-global matching framework, achieving accurate sub-pixel disparities from DP images.

Proposed Method

The CCA methodology commences by fitting parabolas to the matching costs and aggregating the coefficients along image paths. The aggregation is governed by a quadratic constraint, maintaining the quadratic form of the total costs and ensuring disparity smoothness. This leads to a highly efficient propagation scheme, allowing pixel-wise minimization in closed-form. The continuous paradigm facilitates robust multi-scale aggregation, which compensates effectively for the varying PSF across resolutions. This novel strategy results in enhanced disparity estimation, capable of performing comparably well on both DSLR and mobile-phone DP images.

Key Contributions

1. Continuous Cost Aggregation: The introduction of a continuous cost aggregation mechanism offers an efficient solution to disparity extraction challenges inherent in DP images. By using parabolas to model continuous costs, the algorithm allows for precise sub-pixel disparity estimation while minimizing computation.
2. Multi-Scale Disparity Fusion: The algorithm incorporates multi-scale disparity fusion, rectifying errors and bolstering performance in areas with significant depth-of-field variation. This multi-scale approach is instrumental in managing the depth estimation problems posed by varying PSF across different DP devices.
3. Performance Evaluation: Extensive evaluations using DP data from DSLR and phone cameras illustrate the algorithm's state-of-the-art performance. The algorithm achieves high accuracy without the need for extensive datasets or training, unlike many learning-based approaches.

Performance and Evaluation

The performance tests with DP datasets reveal that the proposed CCA method attains state-of-the-art results in disparity estimation. Its adaptability is demonstrated across DSLR and smartphone platforms, underscoring its robustness without the need for device-specific training. Additionally, CCA shows potential suitability for application in conventional stereo images, competing well against existing methods in terms of accuracy and computational efficiency.

Implications and Future Work

The implications of this research extend to various applications in computer vision where depth estimation plays a crucial role, such as autonomous navigation and image manipulation. The technique's device-independent nature and ability to operate without extensive training datasets make it ideal for real-world applications where cross-device compatibility is essential. Future work could explore advanced priors in multi-scale aggregation, enhanced cost functions leveraging deep learning, and higher-order polynomial representations for expanded use cases.

Conclusion

The Continuous Cost Aggregation method introduced offers a compelling advancement in DP disparity extraction, providing a robust, efficient, and application-agnostic solution. It aligns with modern demands for high-fidelity depth estimation across diverse camera systems, bridging an essential gap in computer vision with precision and efficiency.