Towards Robust and Generalizable Lensless Imaging with Modular Learned Reconstruction (2502.01102v1)

Abstract: Lensless cameras disregard the conventional design that imaging should mimic the human eye. This is done by replacing the lens with a thin mask, and moving image formation to the digital post-processing. State-of-the-art lensless imaging techniques use learned approaches that combine physical modeling and neural networks. However, these approaches make simplifying modeling assumptions for ease of calibration and computation. Moreover, the generalizability of learned approaches to lensless measurements of new masks has not been studied. To this end, we utilize a modular learned reconstruction in which a key component is a pre-processor prior to image recovery. We theoretically demonstrate the pre-processor's necessity for standard image recovery techniques (Wiener filtering and iterative algorithms), and through extensive experiments show its effectiveness for multiple lensless imaging approaches and across datasets of different mask types (amplitude and phase). We also perform the first generalization benchmark across mask types to evaluate how well reconstructions trained with one system generalize to others. Our modular reconstruction enables us to use pre-trained components and transfer learning on new systems to cut down weeks of tedious measurements and training. As part of our work, we open-source four datasets, and software for measuring datasets and for training our modular reconstruction.

• The paper proposes a modular reconstruction framework for lensless imaging to improve robustness against noise and model mismatch by dividing processing into components like pre- and post-processors.
• Empirical evaluation shows that this modular approach enhances reconstruction quality and stability against simulated noise and model mismatch compared to conventional methods.
• The study explores generalizability across different imaging systems using a programmable mask system (DigiCam) and finds transfer learning with modular components promising for adapting learned reconstructions.

Robust and Generalizable Lensless Imaging with Modular Learned Reconstruction

The paper by Eric Bezzam, Yohann Perron, and Martin Vetterli provides an explorative paper on the robustness and generalizability of lensless imaging through a modular reconstruction framework. Conventional imaging relies on lenses for clear image capture, but lensless imaging presents an innovative departure from this norm, employing a thin modulating mask and transferring image formation responsibilities to computational algorithms. While beneficial in reducing size and cost, lensless cameras remain challenged by robustness and generalizability issues due to the reliance on simplified physical models and supervised training tailored to specific scenarios.

Modular Reconstruction Framework

The methodology proposed encompasses a modular pipeline with core components such as a pre-processor, camera inversion, post-processor, and PSF correction. This framework is theoretically and experimentally demonstrated to address critical noise amplification and model mismatch transformations identified in conventional imaging recovery techniques. For instance, while traditional approaches suffer significant performance degradation in low SNR environments, the inclusion of a pre-processor mitigates noise before it is magnified through inversion, enhancing the overall reconstruction output.

Empirical Evaluation

Results across multiple datasets with differing imaging systems and mask types display substantial improvements with modular reconstruction. Specifically, metrics such as PSNR, SSIM, and LPIPS indicate enhancements in reconstruction quality when splitting parameters between pre- and post-processors compared to setups relying solely on a post-processor. Furthermore, the experiments reveal superior stability when subjected to simulated shot noise and model mismatch, showcasing the robustness of the proposed methodology.

Generalizability and Transfer Learning

The paper also embarks on assessing the transferability of learned reconstructions across different imaging systems—a concern infrequently addressed in previous literature. The authors introduce DigiCam, a cost-effective programmable-mask system that supports evaluation across multiple PSFs without extensive data collection. The use of transfer learning, whether through fine-tuning on simulated data or adapting pre-learned model components like pre-processors, illustrates promising avenues for application scalability. This work suggests that while adaptation poses challenges under current methodologies, incorporating modular architectures can significantly aid in fostering seamless transitions between varied lensless systems.

Contributions and Future Directions

This research makes a notable contribution by releasing open-source datasets and resources, emphasizing accessibility and reproducibility in lensless imaging research. The exposed limitations in current camera inversion methods under different PSFs underscore the necessity for innovative solutions that transcend conventional image fidelity evaluations.

Future development in lensless imaging could benefit from further exploration of non-linear shift-invariant forward models and contemporary architectures such as transformers or diffusion models. Such innovations could potentially bolster the fidelity of recovered images and extend the practicality of lensless systems in domains where data acquisition is constrained, like medical imaging or privacy-sensitive environments. Overall, this paper proposes a robust framework likely to influence subsequent advancements and applications in lensless computational imaging.