ML-Assisted Holography: Optical & Field Insights

• Machine-learning-assisted holographic models are hybrid frameworks that combine explicit wave-optics simulation with learned components for efficient imaging and inverse design.
• They leverage physics-informed preprocessing and residual neural networks to achieve high localization accuracy, robust phase recovery, and rapid propagation estimation.
• These models accelerate complex optical computations while enabling advancements in 3D particle field imaging and holographic field theory applications.

Searching arXiv for the listed holography-and-ML papers to ground the article and ensure current citation metadata. {"query":"cat:cs.CV OR cat:physics.optics holography machine learning holography 3D particle field imaging (Shao et al., 2019, Wu et al., 2018, Huang et al., 2022, Kavaklı et al., 2021, Ma et al., 2024, Zheng et al., 2024, Shuyang et al., 17 Jun 2025)", "max_results": 10, "sort_by": "relevance"} Reviewing the returned papers for exact arXiv IDs and titles relevant to machine-learning-assisted holographic models. Machine-learning-assisted holographic models are hybrid inverse or forward modeling frameworks in which wave-optical structure remains explicit while machine learning is used to approximate, regularize, accelerate, or invert computationally difficult parts of holographic imaging, computer-generated holography, metasurface design, or holographically motivated field-theoretic constructions. In the applied optics literature, this usually means that diffraction propagation, hologram formation, or calibration remains physics-based, while a neural model performs tasks such as particle localization, autofocusing, phase recovery, transport approximation, inverse design, or few-shot adaptation (Shao et al., 2019, Wu et al., 2018, Huang et al., 2022, Kavaklı et al., 2021). In a more conceptual strand, machine learning is used to construct renormalization-group-like or exact holographic mappings whose learned hierarchy is interpreted as an emergent bulk direction (Howard, 2018, Hu et al., 2019).

1. Scope and definitions

The phrase “machine-learning-assisted holographic model” does not denote a single architecture. In the optical domain, it refers to methods that retain a substantial role for diffraction theory, propagation operators, or instrument calibration while delegating difficult inference or design steps to learned components. Typical targets include 3D particle-field recovery from a single in-line hologram, autofocusing and phase recovery from intensity-only measurements, transfer learning across specimen classes in holographic microscopy, learned forward transport surrogates for computer-generated holography, and inverse design of reconfigurable holographic metasurfaces (Shao et al., 2019, Wu et al., 2018, Huang et al., 2022, Zheng et al., 2024).

A common misconception is that machine-learning-assisted holography is synonymous with end-to-end black-box image synthesis. The literature summarized here shows the opposite pattern: many of the strongest results are obtained when learning is constrained by explicit optics. Physics enters through hologram simulation, angular spectrum or Rayleigh–Sommerfeld propagation, camera-calibrated transport, Born-approximation scattering, or structured target representations, while the network learns the residual mapping from globally entangled diffraction data to sparse particle descriptors, in-focus complex fields, or metasurface control states (Shao et al., 2019, Kavaklı et al., 2021, Ma et al., 2024).

A second usage appears in holographic field theory. There, “holographic model” refers not to optical holography but to bulk-boundary mappings, renormalization, and emergent geometry. In that setting, machine learning assists holography by learning variational RG kernels, exact holographic mappings, or latent structures whose depth is interpreted as a holographic direction (Howard, 2018, Hu et al., 2019).

2. Hybrid optical–ML inverse models

A representative optical example is the reconstruction of sparse 3D particle fields from a single in-line hologram. In “Machine Learning Holography for 3D Particle Field Imaging” (Shao et al., 2019), the inverse problem is to infer particle centroids $(x,y)$ and depth $z$ from dense diffraction patterns in regimes of high concentration, strong cross-interference, and noisy backgrounds. The method is explicitly hybrid. Rather than ingesting only the raw hologram, it uses three channels: the original hologram, a depth projection map, and a maximum phase projection map obtained from angular-spectrum reconstruction. The network output is also task-specific: a grayscale depth channel and a binary centroid channel. This design makes the learned model a sparse descriptor predictor rather than a generic image enhancer.

The same paper shows why this assistance is necessary. Classical pipelines based on propagation plus thresholding or focus metrics degrade rapidly as concentration rises, whereas the proposed modified U-net with residual connections, Swish activation, and channel-specific loss design maintains extraction rates above $94.4\%$ over concentrations from $1.9\times10^{-4}$ to $6.1\times10^{-2}\,\text{ppp}$ (Shao et al., 2019). At fixed concentration $0.018\,\text{ppp}$, it reports $98.7\%$ extraction with ghost particles at $2.3\%$, compared with $40.4\%$ extraction for the Shimobaba deep model and $88.4\%$ for RIHVR (Shao et al., 2019). The main point is structural: the learned component replaces volumetric searching, threshold tuning, and difficult inverse localization, but only after optical priors have been injected through preprocessing.

A related but distinct model appears in “Machine learning holography for measuring 3D particle size distribution” (Shao et al., 2019). Here the objective is not merely point localization but plane-wise recovery of in-focus particle shapes and centroid regions from digitally reconstructed slices. The three-channel input combines a reconstruction slice, the original hologram, and a minimum intensity projection. The two outputs are a particle size map and a centroid map. That output structure is then post-processed into 3D particle position, equivalent diameter, and eccentricity. This suggests a broader pattern: assistance is most effective when the network is aligned with the measurement task, not when it is asked to learn a generic reconstruction.

The HOLODEC pipeline in “Neural network processing of holographic images” (Schreck et al., 2022) reinforces the same lesson. HolodecML does not invert raw holograms end-to-end; it segments reconstructed image tiles and then clusters detections across $z$0. Its most notable methodological result is not architectural novelty but domain-gap control. Synthetic-only training produced excellent synthetic F1 but failed badly on real HOLODEC data, whereas adding corrupting transformations to synthetic data raised HOLODEC tile-level F1 from $z$1 to $z$2 (Schreck et al., 2022). This suggests that in machine-learning-assisted holography, realism of the forward model or of simulation corruption can be as important as network capacity.

The early colloidal-particle study “Machine-learning techniques for fast and accurate feature localization in holograms of colloidal particles” (Hannel et al., 2018) represents an even more modular variant. There, machine learning only performs front-end feature detection and localization, while Lorenz–Mie fitting remains the physics-based back end. The CNN is precise enough to bootstrap Lorenz–Mie inversion, while the Haar cascade emphasizes speed for real-time targeting (Hannel et al., 2018). This is a minimal but influential formulation of machine-learning-assisted holography: learned perception, physics-based quantitative inference.

3. Autofocusing, phase recovery, and transfer learning

In-line holography requires both phase recovery and depth determination, and one of the earliest demonstrations that these steps can be merged into a learned surrogate is “Extended depth-of-field in holographic image reconstruction using deep learning based auto-focusing and phase-recovery” (Wu et al., 2018). HIDEF trains a U-Net-inspired CNN on pairs of randomly defocused back-propagated holograms and in-focus phase-recovered targets. After training, a single forward pass jointly performs autofocus and phase recovery from a single back-propagated hologram. The paper explicitly frames this as a reduction in algorithmic complexity from $z$3 to $z$4, where $z$5 is the number of particles or object points and $z$6 is the focusing search space (Wu et al., 2018).

The technical significance of HIDEF is that it learns a statistical mapping from distorted propagated fields to in-focus complex images rather than numerically reproducing free-space propagation. Its robustness is therefore limited by training defocus range and specimen class. The paper states that outside the trained axial interval the network may hallucinate unrelated sharp-looking features (Wu et al., 2018). This is not merely a caveat; it clarifies the ontology of machine-learning-assisted holography. The learned model absorbs expensive inverse operations, but it does so by exploiting data priors rather than by becoming a universal wave solver.

The transfer problem becomes more explicit in “Few-shot Transfer Learning for Holographic Image Reconstruction using a Recurrent Neural Network” (Huang et al., 2022). There, the underlying task is multi-height holographic reconstruction of the complex field from a small number of intensity-only holograms. The key observation is that the recurrent blocks fusing holograms across axial positions encode comparatively sample-agnostic physics, whereas the convolutional blocks are more sample-dependent. Freezing the GRU-based recurrent backbone and fine-tuning only the convolutional blocks reduces trainable parameters from about $z$7M to $z$8M while preserving adaptation performance (Huang et al., 2022). The reported gains are approximately $z$9-fold faster convergence and approximately $94.4\%$0 lower computation time per epoch (Huang et al., 2022).

This partition between “physics-fusion” and “content-prior” subnetworks is one of the clearest general design principles in the literature. It suggests that machine-learning-assisted holographic models can be made more transferable when their architecture mirrors the physics-versus-content decomposition of the inverse problem. A plausible implication is that similar modular freezing strategies should be effective in other multi-measurement computational imaging systems with explicit propagation or coding diversity.

4. Learned forward models and hologram synthesis

Machine learning can also assist holography on the forward side, by replacing or calibrating the propagation operator used inside hologram optimization. “Learned holographic light transport” (Kavaklı et al., 2021) is exemplary in this regard. Rather than learning holograms directly, it learns a full-resolution complex-valued convolution kernel $94.4\%$1 that maps a phase-only hologram displayed on a real SLM to the camera-captured reconstruction at a fixed plane. This kernel replaces the ideal propagation kernel in a differentiable forward model. The learned model has approximately $94.4\%$2 scalar parameters and achieves train and test L2 losses of $94.4\%$3 and $94.4\%$4, respectively (Kavaklı et al., 2021).

The importance of this work lies in its restraint. The learned component is a single complex transport kernel rather than a large black-box network. It preserves shift-invariant coherent propagation structure while absorbing aberrations, asymmetries, and optical nonidealities of the display (Kavaklı et al., 2021). This makes it both physically interpretable and directly reusable inside inverse CGH optimization. A common misconception is that calibrating a holographic display necessarily requires a large CNN; this paper shows that a learned optical kernel can already reduce the simulation-to-reality gap substantially.

A more display-oriented surrogate appears in “Focal Surface Holographic Light Transport using Learned Spatially Adaptive Convolutions” (Zheng et al., 2024). Standard optimization-based CGH propagates a phase-only SLM field to many planar depth slices, which is costly. This work replaces multiplane propagation with a focal-surface representation and learns a transport operator $94.4\%$5 from hologram and focal-surface map to focal-surface reconstruction. Spatially adaptive convolution is the core mechanism because focal-surface propagation is inherently spatially varying. Against a U-Net baseline, the learned model improves focal-surface transport accuracy from PSNR $94.4\%$6 dB to $94.4\%$7 dB at $94.4\%$8 mm and reduces forward simulation time from $94.4\%$9 s for six-plane ASM to $1.9\times10^{-4}$0 s for the learned transport (Zheng et al., 2024).

The result is not an end-to-end hologram generator. It is a learned differentiable surrogate inside optimization. Final holograms are still evaluated with ASM, and six focal surfaces yield roughly comparable quality to six-plane ASM optimization with about $1.9\times10^{-4}$1 speedup, rounded in the paper to “up to $1.9\times10^{-4}$2” (Zheng et al., 2024). This clarifies an important category within machine-learning-assisted holography: learned forward models that accelerate physics-based optimization without replacing it.

“Time-multiplexed Neural Holography” (Choi et al., 2022) occupies a related calibration-centered position. It addresses heavily quantized phase-only SLMs by combining camera-calibrated wave propagation with quantization-aware optimization and time multiplexing. The propagation model is hybrid: angular-spectrum propagation augmented with learned source-plane and Fourier-plane calibration terms plus CNN modules (Choi et al., 2022). The key contribution is not amortized hologram prediction but differentiable optimization through a calibrated forward model and a surrogate quantizer. This again exemplifies machine-learning assistance as calibration and optimization support rather than end-to-end replacement.

The 2025 CGH literature extends these trends. “MobileHolo” (Shuyang et al., 17 Jun 2025) argues that diffraction demands a flexible effective receptive field and introduces complex-valued deformable convolution within a lightweight U-Net-like network. At $1.9\times10^{-4}$3, it reports PSNR $1.9\times10^{-4}$4 on DIV2K/Flickr2K, improving over CCNN-CGH by $1.9\times10^{-4}$5 dB while using about one-eighth as many parameters (Shuyang et al., 17 Jun 2025). “Unfolding Framework with Complex-Valued Deformable Attention for High-Quality Computer-Generated Hologram Generation” (Zhang et al., 29 Aug 2025) instead embeds learning inside a model-based solver: an ABPM propagation block for wider working distances and a phase-domain complex-valued denoiser with deformable attention. At $1.9\times10^{-4}$6, it reports average PSNR $1.9\times10^{-4}$7 and SSIM $1.9\times10^{-4}$8, outperforming GS, HoloNet, and CCNN-CGH in the reported comparison (Zhang et al., 29 Aug 2025). These works indicate a shift toward architectures that are explicitly aware of phase, long-range coupling, and the limitations of ASM.

5. Inverse design for holographic metasurfaces

In metasurface antennas, the holographic problem is an inverse design task rather than optical image reconstruction. “Deep-learning-assisted reconfigurable metasurface antenna for real-time holographic beam steering” (Ma et al., 2024) formulates a learned inverse map from a desired far-field intensity image to the $1.9\times10^{-4}$9-dimensional control vector of a $6.1\times10^{-2}\,\text{ppp}$0 dipole metasurface. The encoder is a ResNet-based network; the decoder is not learned but is the Born-approximation scattering equation itself (Ma et al., 2024). Training is therefore physics-informed and label-free with respect to optimal state patterns: the supervision is on the far-field pattern, not on hidden control states.

This is a particularly transparent machine-learning-assisted holographic model because the latent code is physically meaningful. The network outputs $6.1\times10^{-2}\,\text{ppp}$1, which determine dipole polarizabilities $6.1\times10^{-2}\,\text{ppp}$2 (Ma et al., 2024). The forward model computes the scattered far field from those states, and loss is an MSE between normalized scattered-field intensity and the target image (Ma et al., 2024). The reported inference times are $6.1\times10^{-2}\,\text{ppp}$3, $6.1\times10^{-2}\,\text{ppp}$4, and $6.1\times10^{-2}\,\text{ppp}$5 for first-, second-, and third-order Born decoders, compared with $6.1\times10^{-2}\,\text{ppp}$6 s for a genetic algorithm and $6.1\times10^{-2}\,\text{ppp}$7 s for modified Gerchberg–Saxton (Ma et al., 2024).

The significance of this metasurface work is that it generalizes the notion of holography beyond beam steering to arbitrary far-field image formation. Because the decoder is a hard-coded scattering model, the method remains interpretable and constraint-aware. A plausible implication is that inverse design in reconfigurable wave systems benefits especially from physics-decoder architectures when hidden control states are expensive to label but the forward Maxwell operator is available.

6. Dataset construction, noise modeling, and conceptual holographic mappings

Machine-learning-assisted holography depends strongly on supervision quality, and recent work has made dataset generation itself part of the modeling problem. “A Large-Depth-Range Layer-Based Hologram Dataset for Machine Learning-Based 3D Computer-Generated Holography” (Lee et al., 24 Dec 2025) introduces KOREATECH-CGH, a public dataset of 6,000 RGB-D/complex-hologram pairs across resolutions from $6.1\times10^{-2}\,\text{ppp}$8 to $6.1\times10^{-2}\,\text{ppp}$9, with depth ranges chosen according to the ASM propagation limit (Lee et al., 24 Dec 2025). The paper’s amplitude projection refinement improves average FIP PSNR from $0.018\,\text{ppp}$0 dB for ADV-LBM to $0.018\,\text{ppp}$1 dB for AP-LBM, with SSIM improving from $0.018\,\text{ppp}$2 to $0.018\,\text{ppp}$3 (Lee et al., 24 Dec 2025). This is important because it reframes machine-learning assistance as a data-generation problem: better optical targets lead to better learned inverse models.

Noise and nonidealities can also be embedded at the measurement stage. “Randomness assisted in-line holography with deep learning” (Manisha et al., 2023) records holograms as second-order intensity-fluctuation correlations under random illumination, then uses an unsupervised auto-encoder with a physics-based consistency loss for twin-image removal. For object $0.018\,\text{ppp}$4, visibility improves from $0.018\,\text{ppp}$5 to $0.018\,\text{ppp}$6, and reconstruction efficiency from $0.018\,\text{ppp}$7 to $0.018\,\text{ppp}$8; for object $0.018\,\text{ppp}$9, visibility improves from $98.7\%$0 to $98.7\%$1, and reconstruction efficiency from $98.7\%$2 to $98.7\%$3 (Manisha et al., 2023). The learned module here is not a standard supervised denoiser but a deep-prior-like reconstruction prior. This suggests a broader category of machine-learning-assisted holography in which learning regularizes numerical inversion after a deliberately modified statistical measurement model.

A conceptually different branch concerns holography in the RG/emergent-geometry sense. “Holographic Renormalization with Machine learning” (Howard, 2018) argues that RBM hidden layers can be interpreted as successive coarse-graining steps and hence as an emergent holographic direction. The core mechanism is the identification

$98.7\%$4

The paper is heuristic and contains no numerical experiments, but it establishes the vocabulary linking deep learning, RG, and holography (Howard, 2018).

“Machine Learning Holographic Mapping by Neural Network Renormalization Group” (Hu et al., 2019) makes this constructive. It trains a hierarchical invertible generative model so that simple bulk Gaussian noise maps to the boundary field ensemble of an interacting complex $98.7\%$5 theory, thereby learning an exact holographic mapping (Hu et al., 2019). The KL objective

$98.7\%$6

is optimized over the invertible map $98.7\%$7, and the emergent bulk geometry is then inferred from residual mutual information (Hu et al., 2019). In the reported application, the learned distance structure is consistent with three-dimensional hyperbolic space $98.7\%$8, i.e. AdS$98.7\%$9 after Wick rotation (Hu et al., 2019). Here machine learning does not assist optical holography at all; it assists holographic model building in strongly coupled field theory.

7. Limitations, controversies, and directions

Across the optical literature, the chief limitation is distribution dependence. HIDEF is explicit that outputs outside the trained defocus interval may hallucinate structures (Wu et al., 2018). Particle-localization models depend on recording parameters, particle statistics, and labeling quality, and often require retraining or transfer learning for new setups (Shao et al., 2019, Shao et al., 2019). HOLODEC shows that ideal simulation alone is insufficient for real data, and that corruption modeling can dominate transfer performance (Schreck et al., 2022). These findings argue against the misconception that a sufficiently large network automatically learns the optics.

A second recurring limitation is that many “fast” learned methods still inherit expensive preprocessing or postprocessing. In the 3D particle-field model, more than $2.3\%$0 of runtime is spent outside network inference (Shao et al., 2019). Time-multiplexed neural holography uses learning mainly in calibration, but hologram generation still takes tens of seconds to minutes (Choi et al., 2022). Learned transport surrogates speed optimization, yet final hologram quality must still be judged under the original physics model (Zheng et al., 2024).

A third limitation concerns generality of the forward model. Learned holographic light transport uses a single shift-invariant kernel at one depth and one wavelength (Kavaklı et al., 2021). The metasurface inverse-design model uses point dipoles, real-valued polarizabilities, and Born approximations, with robustness to model mismatch left open (Ma et al., 2024). ABPM-based unfolding extends propagation range, but still assumes a particular phase-only CGH setup (Zhang et al., 29 Aug 2025). These are not failures so much as evidence that machine-learning-assisted holography works best when its operational envelope is clearly specified.

In the conceptual holography literature, the main controversy is evidential strength. The RBM/RG/AdS analogy in (Howard, 2018) is structural rather than quantitative. The neural exact holographic mapping in (Hu et al., 2019) is more rigorous but still yields an approximate classical bulk representation rather than a full gravity dual. A plausible implication is that machine learning is presently more mature as an assistant to holographic computation and representation design than as an arbiter of holographic duality itself.

Taken together, the field supports several stable conclusions. First, the most effective machine-learning-assisted holographic models are rarely pure black boxes; they are hybrid systems with explicit propagation, calibration, or scattering structure. Second, representation choice matters as much as architecture choice: three-channel physics-assisted inputs, centroid/depth-separated outputs, physically meaningful latent variables, and stage-wise unfolding all improve tractability (Shao et al., 2019, Shao et al., 2019, Ma et al., 2024, Zhang et al., 29 Aug 2025). Third, data realism is decisive. Synthetic supervision succeeds when the simulator, corruption model, or post-processed target distribution matches the instrument sufficiently well (Schreck et al., 2022, Lee et al., 24 Dec 2025). Finally, the term “holographic model” spans both optical engineering and holographic field theory, and machine learning assists each in different ways: by learning inverse or forward operators in the former, and by constructing multiscale or bulk-boundary mappings in the latter (Hu et al., 2019, Howard, 2018).