Resolution-Agnostic Lensless Imaging via Fourier Neural Operators

Abstract: Lensless cameras based on thin diffusers offer a compact alternative to conventional refractive imaging but rely on computational reconstruction, since the diffuser's point spread function (PSF) globally multiplexes every scene point across the sensor. Here, we report a Fourier Neural Operator (FNO) framework for this reconstruction task. Because a linear shift-invariant forward model reduces to a pointwise multiplication in Fourier space, the spectral-domain kernel of an FNO layer is structurally aligned with the DiffuserCam inverse problem. Using a compact DiffuserCam prototype and a 25,000-image natural-scene dataset, our FNO improves upon a U-Net baseline of comparable parameter count by $2.14$~dB in PSNR and $0.11$ in SSIM. The same FNO, trained exclusively at $128 \times 128$, reconstructs $256 \times 256$ and $512 \times 512$ measurements with less than $1$~dB loss in PSNR and no retraining, demonstrating resolution-agnostic inference. The framework is directly applicable to other lensless modalities with global PSFs, such as multimode-fiber endoscopy.

• The paper introduces a Fourier Neural Operator that leverages spectral-domain inversion to enhance image reconstruction fidelity and efficiency.
• The methodology utilizes FFT-based global filtering to overcome the limitations of local CNN kernels, achieving higher PSNR and SSIM compared to U-Net.
• The study validates resolution-agnostic inference by demonstrating accurate reconstructions at varying resolutions with minimal degradation.

Resolution-Agnostic Lensless Imaging via Fourier Neural Operators

Introduction

Lensless computational imaging systems, exemplified by the DiffuserCam architecture, exploit thin diffusers to replace traditional refractive optics, enabling highly compact designs. However, these systems inherently require sophisticated computational methods for image reconstruction, as the point spread function (PSF) of the diffuser multiplexes scene information globally over the sensor. Classical iterative approaches integrating hand-crafted priors are effective but resource-intensive. Recent advances leverage deep convolutional neural networks (CNNs), such as U-Net architectures, for data-driven end-to-end inversion. These methods, while performant, are constrained by the locality of convolutional kernels and fixed input resolutions, presenting challenges in generalization across varying discretizations and modalities with inherently global PSFs.

The discussed work proposes and rigorously evaluates a Fourier Neural Operator (FNO) architecture aligned with the inherent spectral characteristics of the lensless imaging inverse problem. The FNO framework demonstrates significant improvements in both reconstruction fidelity and practical utility, notably supporting resolution-agnostic inference.

DiffuserCam System and Dataset Acquisition

The experimental setup employs a dual-aperture DiffuserCam, where a tablet displays natural-scene images that are simultaneously captured by a standard camera and a diffuser-coated sensor for ground-truth/reference and measurement data, respectively (Figure 1).

Figure 1: Experimental setup of the DiffuserCam with dual optical paths for simultaneous ground-truth and diffuser-based data acquisition.

This design yields precise alignment between raw lensless measurements and ground-truth reconstructions, ensuring high-quality paired data. A diverse set of 25,000 images from the MIR-Flickr dataset is employed, downsampled to multiple resolutions ($512 \times 512$, $256 \times 256$, $128 \times 128$) for evaluating discretization invariance.

Fourier Neural Operator Architecture

The core FNO model comprises an input lifting layer, followed by six sequential Fourier layers and a final projection layer. Each Fourier layer executes a learned spectral filtering operation:

1. Input features are lifted to a higher channel dimension.
2. A 2D FFT is applied to yield spectral representations.
3. Complex-valued learnable filters are multiplied in frequency space, implementing a global convolution.
4. An inverse FFT reconstructs spatial domain features.
5. A $1 \times 1$ convolution bypass preserves local information, and GELU nonlinearity is applied.

This architecture ensures that the convolutional inversion of lensless measurements is conducted through a global operation directly suitable for the system's physics (Figure 2).

Figure 2: The FNO architecture, processing lensless measurements in Fourier space to capture global dependencies and maintain resolution agnosticism.

In the evaluated instance, the FNO retains $24 \times 24$ spectral modes, corresponding to the lowest 37% of the spectrum for $128 \times 128$ resolution. This mode selection is calibrated to match the network's parameter count with a baseline U-Net for equitable comparison.

Comparative Evaluation: FNO vs. U-Net

On the $128 \times 128$ test set, the FNO surpasses the U-Net baseline by a significant margin: achieving an average PSNR of 22.32 dB (+2.14 dB over U-Net) and an SSIM improvement of 0.11. Additionally, FNO converges more rapidly and with reduced GPU memory consumption.

Figure 3: Representative reconstructions where FNO demonstrates consistent advantage in PSNR and SSIM over the U-Net for varying image complexities.

The advantage is attributed to the FNO's ability to globally invert the system's multiplexing PSF, in contrast to the U-Net which relies on increasingly broad receptive fields via local convolutions and skip connections. The experimental results rigorously demonstrate the superiority of operator-theoretic, spectral-domain architectures for lensless imaging governed by linear, shift-invariant global scattering.

Resolution-Agnostic Inference

A central claim is that FNOs, trained exclusively on $128 \times 128$ measurements, can natively accept and accurately process higher resolution ($256 \times 256$ and $512 \times 512$) inputs without any architectural modification or retraining. Quantitative metrics indicate less than 1 dB degradation in PSNR (21.35 dB at $256 \times 256$0 vs. 22.32 dB at $256 \times 256$1), consistently preserving SSIM.

Figure 4: Demonstration of FNO generalization—model trained at $256 \times 256$2 infers on $256 \times 256$3 inputs without retraining, preserving reconstruction quality despite previously unseen spatial frequencies.

Resolution-agnostic generalization validates that neural operators learn mappings between function spaces rather than discrete arrays, encoding system physics so that sampling density can increase at inference time, unlike conventional CNNs. This property is theoretically underpinned by the alignment of the FNO layer’s action with the convolution theorem and spectral inversion of the PSF.

Broader Implications, Limitations, and Outlook

The success of the spectral-domain approach is not limited to the specific diffuser-based lensless configuration, but extends to any system where the forward operator is a global convolution (including computational microscopy through scattering, multimode-fiber endoscopy, and single-pixel cameras). For practical deployment, the efficiency and memory advantages are pronounced, especially for embedded or resource-constrained imaging devices.

However, the comparison is limited to a purely data-driven U-Net baseline. Hybrid model-based methods incorporating explicit PSF priors (e.g., FlatNet, Le-ADMM-Net) might provide further improvements but were not evaluated side-by-side. The current dataset is based on planar tablet images and may not fully reflect depth-encoded or more complex 3D scenes.

Future directions involve explicit physical PSF incorporation into the neural operator framework, comparison with advanced hybrid models, and comprehensive robustness studies under real-world 3D scenes and noise conditions.

Conclusion

This study demonstrates that Fourier Neural Operators, structurally aligned with the global, shift-invariant nature of lensless imaging PSFs, substantially improve the fidelity and flexibility of lensless image reconstruction. The FNO achieves strong numerical gains over CNN baselines and, critically, supports resolution-agnostic inference—an attribute unattainable by standard pixel-grid neural networks. These advances highlight the efficacy of operator-theoretic deep learning for inverse problems in computational imaging, setting a precedent for physics-aligned, resolution-flexible methodologies in the field.