Fovea Stacking: Imaging with Dynamic Localized Aberration Correction (2506.00716v1)

Abstract: The desire for cameras with smaller form factors has recently lead to a push for exploring computational imaging systems with reduced optical complexity such as a smaller number of lens elements. Unfortunately such simplified optical systems usually suffer from severe aberrations, especially in off-axis regions, which can be difficult to correct purely in software. In this paper we introduce Fovea Stacking, a new type of imaging system that utilizes emerging dynamic optical components called deformable phase plates (DPPs) for localized aberration correction anywhere on the image sensor. By optimizing DPP deformations through a differentiable optical model, off-axis aberrations are corrected locally, producing a foveated image with enhanced sharpness at the fixation point - analogous to the eye's fovea. Stacking multiple such foveated images, each with a different fixation point, yields a composite image free from aberrations. To efficiently cover the entire field of view, we propose joint optimization of DPP deformations under imaging budget constraints. Due to the DPP device's non-linear behavior, we introduce a neural network-based control model for improved alignment between simulation-hardware performance. We further demonstrated that for extended depth-of-field imaging, fovea stacking outperforms traditional focus stacking in image quality. By integrating object detection or eye-tracking, the system can dynamically adjust the lens to track the object of interest-enabling real-time foveated video suitable for downstream applications such as surveillance or foveated virtual reality displays.

Fovea Stacking: Imaging with Dynamic Localized Aberration Correction

The paper "Fovea Stacking: Imaging with Dynamic Localized Aberration Correction" outlines an innovative approach to computational imaging that leverages deformable phase plates (DPPs) for aberration correction, aiming to optimize the optical design of compact camera systems. The primary challenge addressed by the authors is the correction of aberrations in imaging systems with reduced optical complexity, particularly in off-axis regions.

Technical Approach and System Design

The proposed system introduces Fovea Stacking, a technique that employs DPPs to dynamically correct localized optical aberrations. This is achieved by optimizing DPP deformations through a differentiable optical model. The term "fovea" refers to a localized region with enhanced image sharpness that can be dynamically moved across the sensor plane. Images captured at different fixation points are stacked to produce a composite image devoid of aberrations. An effective field of view (FoV) coverage is achieved using joint optimization methods to minimize the required number of images.

The system architecture consists of an achromatic doublet lens and a refractive DPP, specifically chosen for its capability to modulate the wavefront by altering local surface geometries. Unlike traditional adaptive optics that utilize reflective deformable mirrors, the DPP offers a compact, transmissive alternative suited for integration into small devices.

Model and Optimization

The DPP is characterized using Zernike polynomials, enabling precise control over localized wavefront corrections. The authors utilize a forward model for ray tracing through the optical system to calculate point spread functions (PSFs) and perform realistic simulations. This model is implemented in a differentiable manner, facilitating optimizations necessary for high fidelity imaging. The optimization focuses on minimizing the PSF radius to improve image quality, and wavefronts are jointly optimized to cover the full FoV efficiently using just a few images.

Neural Network-Based Control Model

To address the non-linear behaviors of DPPs, particularly for larger control signals, the authors developed a neural network-based control model. This model maps desired Zernike coefficients to electrode voltages, improving alignment between simulation results and actual hardware performance. Experiments demonstrate the superior accuracy of this control strategy compared to linear models.

Experimental Validation and Applications

The paper provides detailed experimental validation using a prototype system, demonstrating:

• Aberration-Corrected Imaging: Fovea stacking yields higher quality images compared to traditional focus stacking methods by addressing off-axis aberrations more effectively.
• Extended Depth-of-Field Imaging: By stacking images captured at various depths, the system improves focus across a range of distances, outperforming conventional methods.
• Foveated Object Tracking: Integrated with object detection or eye tracking, the system dynamically adjusts the imaging focus to maintain areas of interest sharp within the fovea region.

Implications and Future Directions

The implications of this research span both practical and theoretical domains. Practically, the use of DPPs represents a step towards miniaturizing high-performance optical systems suitable for mobile devices, potentially superseding reflective wavefront modulators. Theoretically, this research paves the way for novel optical designs that blend optics and computing, facilitating adaptive optical systems.

Future work could involve further miniaturization of DPP devices, enabling widespread adoption in consumer electronics. Real-time dynamic control and scene-dependent imaging adjustments could be explored to enhance system responsiveness and versatility.

In conclusion, this paper proposes a significant advancement in the field of computational imaging, utilizing dynamic localized aberration correction to overcome inherent limitations in traditional optical systems. The novel combination of DPP technology with advanced optimization techniques presents new opportunities for both theoretical exploration and practical applications in compact, high-performance camera systems.