LearnLens: Learning-Based Lensless Imaging

• LearnLens is a learning-based lensless imaging system that integrates PSF estimation with image reconstruction using an end-to-end deep network.
• It replaces traditional calibration pipelines with joint spatial and frequency domain optimization to enhance high-frequency detail recovery and robustness.
• The architecture employs specialized modules—SCM, CMS, WNFB, and SAM—to extract multi-scale features and perform FFT-based deconvolution with learned PSF surrogates.

LearnLens is a learning-based lensless imaging system that learns the lens/PSF and reconstructs images end-to-end. In the formulation associated with LensNet, it addresses a central limitation of lensless cameras: image formation is governed by the point spread function (PSF), yet conventional pipelines often rely on explicit calibration, fixed or approximate PSF models, iterative solvers, and hand-crafted pre-processing. LearnLens replaces that separation between optics and inversion with joint optimization of PSF estimation and reconstruction, combining spatial-domain feature extraction with frequency-domain deconvolution inside one trainable pipeline (Bai et al., 3 May 2025).

1. Problem setting and motivation

Lensless imaging uses an optical encoder, such as a diffuser or coded mask, near the sensor to modulate incoming light and then depends on computational reconstruction. The captured signal is governed by the PSF, which describes how a point source is distributed on the sensor. Traditional pipelines depend on explicit calibration or fixed/approximate PSF models, often with iterative solvers such as ADMM and FISTA and with hand-crafted pre-processing. These static models struggle with model mismatch caused by changes in scene depth, illumination, pose, and minor misalignments; with sensor noise and manufacturing imperfections; and with large PSF support, which introduces strong global multiplexing, makes local details hard to recover, and increases memory and computation overhead for inversion (Bai et al., 3 May 2025).

The LearnLens premise is that end-to-end, data-driven PSF estimation is beneficial because the network can adapt the PSF to real data without repeated calibration, can learn real-world imperfections such as mask nonidealities, misalignment, and spectral or sensor effects from measurements, and can jointly optimize PSF estimation and reconstruction to reduce reliance on hand-tuned priors while improving high-frequency recovery and noise attenuation. This places LearnLens within the broader lensless-imaging paradigm in which the lens is removed and computation replaces optical focusing; open platforms such as LenslessPiCam make this paradigm experimentally accessible, but still foreground calibration and inverse-problem design as major practical concerns (Bezzam et al., 2022).

2. Optical model and key formalism

The central optical quantity in LearnLens is the PSF, written as $h(x,y)$ or $\mathrm{PSF}(x,y)$, defined as the intensity response of the system to a unit point source. Its Fourier transform is the optical transfer function,

$H(\omega_x,\omega_y)=\mathcal{F}\{h(x,y)\},$

with modulation transfer function $\lvert H\rvert$ and phase transfer function $\arg(H)$. Under a shift-invariant approximation, the spatial-domain forward model is

$y(x,y)=x(x,y)*h(x,y)+n(x,y),$

where $x$ is the object, $y$ the measurement, $h$ the PSF, $n$ noise, and $\mathrm{PSF}(x,y)$0 convolution. In coded-aperture systems at fixed geometry, the more general linear model $\mathrm{PSF}(x,y)$1 is often used; when the mask and scene satisfy a shift-invariance approximation, $\mathrm{PSF}(x,y)$2 reduces to convolution with $\mathrm{PSF}(x,y)$3. The corresponding frequency-domain form is

$\mathrm{PSF}(x,y)$4

These definitions determine both the inversion problem and the numerical stability limits of any reconstruction method (Bai et al., 3 May 2025).

LearnLens adopts this formalism but does not treat the PSF as a static calibration artifact. Instead, it learns a PSF-like surrogate from the measurement stream itself. A plausible implication is that LearnLens belongs to the same broader family of PSF-aware methods as continuous blur-field modeling, where a single MLP maps image-plane location, focus setting, and optionally depth to a spatially varying PSF; that line of work emphasizes continuous parameterization, whereas LearnLens emphasizes end-to-end inversion inside a lensless reconstruction network (Lin et al., 2023).

3. Architecture and computational pipeline

LensNet, which operationalizes the LearnLens formulation, is an end-to-end encoder–decoder network that unifies spatial-domain processing with frequency-domain deconvolution. Multi-scale spatial features are extracted from the measurement by a Spatial Compression Module (SCM); a learnable Coded Mask Simulator (CMS) infers a PSF-like representation; a Wiener Fusion Block (WNFB) performs FFT-based deconvolution using the learned PSF; and a Spatial Amplification Module (SAM) fuses frequency-refined features with spatial features to reconstruct the image (Bai et al., 3 May 2025).

Module	Function	Mechanism
SCM	Multi-scale spatial encoding	Reconstruction Blocks and gating
CMS	Learned coded-mask/PSF surrogate	Global average pooling and $\mathrm{PSF}(x,y)$5 convolution attention
WNFB	Frequency-domain deconvolution	FFT/IFFT with Wiener-like transfer function
SAM	Spatial-frequency fusion and decoding	Upsampling, concatenation, and reconstruction

The CMS is designed to learn a data-driven surrogate for the PSF induced by the coded mask and optics, including directional modulation and global multiplexing patterns. Operating on feature maps $\mathrm{PSF}(x,y)$6, it uses global average pooling and pointwise convolution to generate channel attention weights:

$\mathrm{PSF}(x,y)$7

All operations are differentiable, so gradients backpropagate through the CMS and downstream FFT blocks. The paper further notes that practitioners can add constraints such as non-negativity through softplus, energy normalization through $\mathrm{PSF}(x,y)$8, or bandlimiting penalties in frequency space to stabilize learning and reflect physics (Bai et al., 3 May 2025).

The WNFB embeds a Wiener-like deconvolution directly in frequency space. Classical Wiener filtering is written as

$\mathrm{PSF}(x,y)$9

while the LensNet instantiation uses

$H(\omega_x,\omega_y)=\mathcal{F}\{h(x,y)\},$0

where $H(\omega_x,\omega_y)=\mathcal{F}\{h(x,y)\},$1 is a small stabilizer. The role of $H(\omega_x,\omega_y)=\mathcal{F}\{h(x,y)\},$2 is to prevent division by near-zero $H(\omega_x,\omega_y)=\mathcal{F}\{h(x,y)\},$3 and to control noise amplification in weak passbands. LensNet uses a fixed $H(\omega_x,\omega_y)=\mathcal{F}\{h(x,y)\},$4, although the framework allows learned or frequency-dependent variants (Bai et al., 3 May 2025).

4. Training procedure, datasets, and empirical performance

LearnLens is trained end-to-end with a composite loss

$H(\omega_x,\omega_y)=\mathcal{F}\{h(x,y)\},$5

where

$H(\omega_x,\omega_y)=\mathcal{F}\{h(x,y)\},$6

The training loop backpropagates through FFT/IFFT, CMS attention, and all reconstruction blocks. In operational terms, the measurement $H(\omega_x,\omega_y)=\mathcal{F}\{h(x,y)\},$7 passes through the SCM to produce spatial features; the CMS outputs a PSF surrogate; FFT yields both PSF-domain and feature-domain representations; the Wiener transfer function refines features in frequency space; and the SAM fuses spatial and frequency outputs into the final reconstruction (Bai et al., 3 May 2025).

Two datasets are central to the reported experiments. DiffuserCam contains 25,000 paired images, preprocessed to $H(\omega_x,\omega_y)=\mathcal{F}\{h(x,y)\},$8 with 24k training and 1k test samples. MWDNs contains 25,000 monitor-displayed images from Kaggle Dogs vs. Cats, preprocessed to $H(\omega_x,\omega_y)=\mathcal{F}\{h(x,y)\},$9. Implementation uses PyTorch, $\lvert H\rvert$0 NVIDIA RTX 3090, batch size 16, the Adam optimizer, and random rotation/flipping augmentations. Baselines include classical or iterative methods—Wiener, vanilla GD, Nesterov GD, FISTA, ADMM, APGD—and deep methods—TikNet, FlatNet, LenslessGAN, UDN, and MWDN. Evaluation uses PSNR, SSIM, and LPIPS (Bai et al., 3 May 2025).

Dataset	LensNet result	Setup
DiffuserCam	PSNR $\lvert H\rvert$1 dB, SSIM $\lvert H\rvert$2, LPIPS $\lvert H\rvert$3	24k train / 1k test, $\lvert H\rvert$4
MWDNs	PSNR $\lvert H\rvert$5 dB, SSIM $\lvert H\rvert$6, LPIPS $\lvert H\rvert$7	25,000 images, $\lvert H\rvert$8

These results are reported as state-of-the-art. Qualitatively, LensNet preserves sharp edges and fine textures better than FlatNet, TikNet, and UDN, while avoiding ringing and oversmoothing common in classical deconvolution. The integrated Wiener step is reported to help recover high-frequency structure and attenuate noise, including subtle textures such as fur and fabrics and scenes with complex lighting. On MWDNs input size, LensNet uses approximately 31.18M parameters and 115.10G FLOPs, compared with approximately 1.04M and 2.37G for UDN, approximately 59.13M and 220.42G for FlatNet/TikNet, approximately 11.77M and 13.82G for LenslessGAN, and approximately 21.96M and 83.85G for MWDN (Bai et al., 3 May 2025).

5. Robustness, implementation, and relation to adjacent methods

LearnLens is designed as a calibration-lite system. Because the CMS learns PSF and mask behavior from data, it can absorb nonuniform mask transmission and slight misalignments, while the WNFB stabilizer $\lvert H\rvert$9 mitigates noise blow-up in weak passbands. End-to-end training with diverse scenes increases robustness; the learned PSF adapts to measurement statistics, and frequency-domain fusion makes the network less sensitive to global multiplexing. The data-driven CMS also reduces dependence on a specific calibrated kernel, improving portability across different masks, distances, and illumination conditions. The implementation guidance therefore recommends avoiding explicit PSF calibration during deployment, and instead fine-tuning the CMS with a small dataset for new devices if necessary. It also recommends mixed precision to reduce FFT cost, caching per-scale FFT shapes, and gradient clipping around frequency operations (Bai et al., 3 May 2025).

Ablation results clarify where the gains arise. Using an original fixed PSF is inferior to the learnable CMS, with lower PSNR and SSIM and higher LPIPS. Removing the WNFB reduces high-frequency recovery and increases artifacts. Reducing network depth through fewer downsampling layers or removing Reconstruction Blocks harms structural coherence and texture preservation. PSF initialization is reported as not critical because of attention-based learning, while regularization strength, if added, must be tuned carefully to avoid oversmoothing (Bai et al., 3 May 2025).

This design is adjacent to, but distinct from, other physics-aware learning programs in computational imaging. Unrolled primal-dual networks for lensless cameras embed learnable forward and adjoint models in a learned primal-dual optimization framework and report reconstruction improvements of up to $\arg(H)$0 dB PSNR compared to methods that do not correct model error (Kingshott et al., 2022). LenslessPiCam, by contrast, emphasizes accessible hardware and modular reconstruction software built around a Raspberry Pi HQ camera, RGB capture, and FISTA, ADMM, and APGD implementations (Bezzam et al., 2022). This suggests that LearnLens occupies a convergent design space in which calibration, forward modeling, and inversion are increasingly absorbed into jointly optimized, physically structured learning systems.

6. Assumptions, failure modes, and future directions

LearnLens retains several simplifying assumptions. Its default model assumes a shift-invariant PSF and linear image formation, so performance may degrade under strong depth variation, non-stationary PSFs, or coherent effects beyond the model. Extremely low SNR and near-zero OTF bands can still cause frequency-amplification artifacts; the stabilizer $\arg(H)$1 mitigates these failures but may also limit deblurring strength. These are not minor implementation details but structural limits of the current formulation (Bai et al., 3 May 2025).

The research agenda described around LearnLens focuses on non-stationary PSF modeling, including space-variant kernels and blind deconvolution with spatially varying CMS outputs; physics-informed constraints such as non-negativity, energy conservation, bandlimiting, and Fresnel phase priors; self-supervised or unsupervised training when paired ground truth is scarce; and multi-spectral and 3D extensions such as depth-aware PSFs and diffuser-based volumetric reconstruction. The paper also identifies learned priors, including diffusion priors applied after frequency fusion, as a route to further perceptual gains. A plausible implication is that future LearnLens systems may converge toward continuous blur-field parameterizations of the sort used in neural lens blur field modeling, where PSFs are represented as functions of image-plane location, focus setting, and optionally depth rather than as implicitly learned attention maps alone (Lin et al., 2023).

In that sense, LearnLens marks a specific stage in the evolution of lensless imaging: from fixed-kernel inversion and heavy calibration toward jointly learned optical surrogates and reconstruction operators. Its defining claim is not merely that deep learning can post-process lensless measurements, but that the PSF itself can become a trainable object inside the reconstruction network, allowing one encoder–decoder pipeline to mediate between physical optics, numerical deconvolution, and practical deployment in applications ranging from miniature sensors to medical diagnostics (Bai et al., 3 May 2025).