Machine vision with small numbers of detected photons per inference

Abstract: Machine vision, including object recognition and image reconstruction, is a central technology in many consumer devices and scientific instruments. The design of machine-vision systems has been revolutionized by the adoption of end-to-end optimization, in which the optical front end and the post-processing back end are jointly optimized. However, while machine vision currently works extremely well in moderate-light or bright-light situations -- where a camera may detect thousands of photons per pixel and billions of photons per frame -- it is far more challenging in very low-light situations. We introduce photon-aware neuromorphic sensing (PANS), an approach for end-to-end optimization in highly photon-starved scenarios. The training incorporates knowledge of the low photon budget and the stochastic nature of light detection when the average number of photons per pixel is near or less than 1. We report a proof-of-principle experimental demonstration in which we performed low-light image classification using PANS, achieving 73% (82%) accuracy on FashionMNIST with an average of only 4.9 (17) detected photons in total per inference, and 86% (97%) on MNIST with 8.6 (29) detected photons -- orders of magnitude more photon-efficient than conventional approaches. We also report simulation studies showing how PANS could be applied to other classification, event-detection, and image-reconstruction tasks. By taking into account the statistics of measurement results for non-classical states or alternative sensing hardware, PANS could in principle be adapted to enable high-accuracy results in quantum and other photon-starved setups.

• The paper presents the photon-aware neuromorphic sensing (PANS) framework, which jointly optimizes the optical encoder and digital decoder to overcome low-photon detection challenges.
• It models the single-photon detection process as a binary stochastic neuron using Poisson statistics, with gradient damping via e^(-lambda) to ensure unbiased learning.
• The study demonstrates that PANS achieves superior accuracy and robustness in ultra-low photon regimes compared to conventional imaging paradigms through task-adapted sparse illumination patterns.

Machine Vision with Small Numbers of Detected Photons per Inference

Introduction and Photon-Aware Neuromorphic Sensing (PANS) Framework

The paper introduces and systematically analyzes photon-aware neuromorphic sensing (PANS), a machine learning framework that is end-to-end optimized directly accounting for the stochasticity and irreversibility of the single-photon detection (SPD) process. The motivation lies in the photon-starved regime, where classical detection bottlenecks dominate and irreversibly discard most information prior to digital post-processing—posing fundamental limits for low-light machine vision, rapid cytometry, and non-invasive biomedical imaging.

The core conceptual advance is the integration of the physical stochastic detection process into the differentiable computational graph, using a custom straight-through estimator with theoretically justified gradient damping. This permits gradient flow even at a detection bottleneck, enabling the joint optimization of programmable optical front-ends (encoders) and arbitrary digital back-ends (decoders). The treatment generalizes to both active and passive optical systems, as well as both coherent and incoherent light.

SPD Modeling, Gradient Estimation, and Training Methodology

The SPD process is formalized as a binary stochastic neuron governed by Poisson statistics. Each output is a Bernoulli trial with success probability $P_{\text{SPD}}(\lambda) = 1 - e^{-\lambda}$ for mean photon flux $\lambda$. The key is to use $e^{-\lambda}$ as the gradient scaling factor during optimization. This ensures unbiased gradient estimation with respect to expected activations, allowing effective training even when activations are heavily quantized/stochastic.

For the optical encoder, both intensity-based and field-based encodings are developed. The former leads naturally to a non-negativity constraint on weights, which is elegantly enforced via squared parameterization instead of clamping/projection, thus preserving gradient flow (demonstrated throughout the training analysis).

Initialization and training stabilization are handled by Kaiming scaling, slope annealing, and post-detection batch normalization. The forward pass remains stochastic; during inference, multiple physical shots can be averaged ($K$-shot scaling) to reduce uncertainty.

Detection Bottleneck and Taxonomy of Paradigms

The detection bottleneck is rigorously framed as an irreversible information-theoretic constraint. The taxonomy constructed in the paper organizes optical sensing approaches into four regimes:

1. Direct Imaging: No encoder, no task optimization, no photon-awareness.
2. Fixed Optical Transformations: Encoder present but not optimized, no photon-awareness.
3. Conventional End-to-End (E2E): Joint encoder-decoder optimization, but deterministic or crude model of detection.
4. PANS (Photon-Aware Neuromorphic Sensing): Full joint optimization including the physical stochastic detection process.

A flowchart of paradigm relationships is given, with PANS emerging as the only physically consistent and performant option when the detection bottleneck is dominant.

Training Dynamics and Efficient Illumination Patterns

The training section details how the illumination regularization $\alpha$ trades off photon budget and accuracy, with sharp transitions between photon-efficient and photon-inefficient solutions. Models are evaluated with both small and large $d_{\text{f}}$ (feature dimension), making clear that aggressive photon regularization induces highly sparse, localized patterns, where the encoder adaptively allocates photons to maximize information retention.

Figure 1: Training dynamics under different illumination regularization $\alpha$ for $d_\text{f} = 10$.

Figure 2: Training dynamics under different illumination regularization $\alpha$ for $d_\text{f} = 16$.

Figure 3: Training dynamics under different illumination regularization $\lambda$0 for $\lambda$1.

Figure 4: Training dynamics under different illumination regularization $\lambda$2 for $\lambda$3.

Figure 5: Evolution of learned illumination patterns during training. Each panel displays the $\lambda$4 patterns, showing spatial sparsity and efficient concentration of photon budget with training progression.

Scaling Analysis: Feature Dimension, $\lambda$5-shot, and Photon Robustness

The analysis demonstrates that—when constrained by the total number of SPD events (detected photons)—scaling up the feature dimension $\lambda$6 while keeping $\lambda$7 (single shot per feature) is more efficient than reusing $\lambda$8 shots per feature. This is proven both theoretically and numerically, with systematic evaluation on the FashionMNIST benchmark. Robustness analyses (to photon-level scaling $\lambda$9 and dark counts) show high tolerance to common experimental impairments, validating the practical deployability.

Figure 6: Multi-shot evaluation validates $e^{-\lambda}$0 scaling over $e^{-\lambda}$1 scaling.

Figure 7: Effect of photon level scaling $e^{-\lambda}$2. Test accuracy versus photon scaling parameter $e^{-\lambda}$3 for trained PANS models with different $e^{-\lambda}$4.

Figure 8: Robustness to dark counts. Test accuracy versus dark count rate for different $e^{-\lambda}$5.

Benchmarking and Comparative Evaluation

Direct imaging in the photon-starved regime is shown to lose all class-discriminative information at $e^{-\lambda}$6. This is qualitatively showcased with shot-noise-limited frames and t-SNE embeddings, illustrating catastrophic information loss.

Figure 9: Shot-noise-limited frames at different photon budgets; object identity is lost in the few-photon regime.

Figure 10: t-SNE visualization of information loss; class clusters collapse to randomness at low photon budget.

Compressive sensing with random illumination patterns provides no improvement in this regime Figure 11. Conventional E2E yields partial recovery but breaks down in the ultra-low photon count regime Figure 12, confirming the necessity of the photon-aware approach.

Figure 11: Compressive sensing performance. Test accuracy as a function of number of patterns ($e^{-\lambda}$7) and photon budget ($e^{-\lambda}$8).

Figure 12: Conventional end-to-end optimization results. Test accuracy versus $e^{-\lambda}$9 for different $K$0.

Generalization: Applications and Real-World Tasks

PANS is experimentally validated in a real optical system with active programmable structured illumination and low-noise digital single-photon detection. The experimental platform includes an OLED for pattern generation, SLM for object modulation, and qCMOS operating in photon-counting mode.

Figure 13: Experimental apparatus for active PANS with OLED illumination, SLM-based object, and qCMOS single-photon camera.

Extensive numerical and experimental results are provided on canonical vision benchmarks (FashionMNIST, MNIST), cytometry (cell-organelle classification, Figure 14), machine reading (barcode detection, Figure 15), and advanced tasks in passive sensing: multimode fiber imaging Figure 16, transient event detection Figure 17, tissue blood flow Figure 18, nebula classification Figure 19, and fiber end-face inspection Figure 20. In all cases, PANS achieves accurate discrimination far below the photon budgets required by prior methods.

Figure 14: Cell-organelle classification dataset: representative images in three biological categories.

Figure 15: Barcode “1010” identification dataset, with samples and position-invariant detection requirement.

Figure 16: Image sensing through multimode fiber: original MNIST digit, MMF propagation, speckle pattern, PANS encoding, and reconstruction.

Figure 17: Transient event detection dataset: background vs. transient-present examples with overlapping means.

Figure 18: Tissue speckle imaging: distinction between static (sharp) and perfused (blurred) tissue.

Figure 19: Nebula classification: planetary nebulae versus emission nebulae; class differences are purely morphological.

Figure 20: Fiber end-face inspection: clean vs. contaminated fibers, highlighting subtle localized anomalies.

Technical Implications and Prospects

This work establishes that, under strict photon budgets, detection noise cannot be mitigated by post-detection processing—even arbitrarily deep neural architectures are fundamentally limited by the pre-detection information loss. The PANS optimization allows the system to learn efficient, task-adapted modes for information preservation, balancing the trade-off between photon use and task fidelity in a physically meaningful, highly regularized manner.

End-to-end training that is photon-aware (as opposed to merely noise-robust or quantization-aware) induces sparse, adaptive, possibly highly nonlinear encoder patterns, and this is essential for operating at the limit where far fewer photons are used than the pixel/mode count. The demonstrated robustness to intensity fluctuations and dark counts ensures experimental viability. Robustness to hardware noise sources and generalization across downstream tasks indicate relevance for ultra-low-light imaging, rapid-high-throughput flow cytometry, covert LIDAR/autonomous driving under adverse weather, and clinical/biomedical photodamage constraints.

Conclusion

The paper presents a rigorous, physically consistent framework for machine vision operating at the detection bottleneck, characterized by joint optical–digital co-optimization with explicit stochastic modeling of the photon counting process. Numerical and experimental evidence supports quantitative improvements in accuracy per detected photon over all baseline paradigms. The PANS methodology, being encoder-agnostic, is extensible to novel photonic materials and quantum-enhanced sensors, making it a principal reference point for optical AI implementations operating near the quantum limit.

(2603.23974)