Photon-Aware Neuromorphic Sensing (PANS)

• Photon-Aware Neuromorphic Sensing is a framework that unifies optical physics, machine learning, and neuromorphic engineering to enable efficient sensing in photon-starved environments.
• It employs end-to-end optimization by integrating photon detection into both optical encoding and digital inference stages, mitigating shot noise and information loss.
• Applications include ultra-low-light imaging, quantum metrology, and event-driven computer vision, demonstrating high accuracy with minimal photon budgets.

Photon-Aware Neuromorphic Sensing (PANS) unifies principles from optical physics, machine learning, and neuromorphic engineering to achieve highly efficient sensing and inference with minimal photon budgets. By integrating the stochastic physics of photon detection directly into end-to-end system optimization—both in algorithmic design and at the device level—PANS enables robust performance in scenarios previously dominated by shot noise and information loss, such as low-light imaging, quantum metrology, and event-driven computer vision.

1. Motivation and Conceptual Foundations

Photon-Aware Neuromorphic Sensing arises from the recognition that in photon-starved regimes—those where the per-inference detected photon count is near or below unity per channel—traditional sensing and machine vision pipelines, which assume continuous-valued, high-SNR detector outputs, become fundamentally inadequate. In these conditions, the discrete and stochastic nature of single-photon detection sets an inescapable information bottleneck: information not captured at the detection stage cannot be recovered downstream, irrespective of neural network sophistication (Ma et al., 25 Mar 2026).

PANS treats the process of photon detection itself (single-photon avalanche, phototransistor response, neuromorphic event) as the central computational primitive. Rather than separating sensing from inference, PANS leverages end-to-end, differentiable architectures or hardware co-design to jointly optimize optical encoding, device physics, and neuromorphic post-processing, maximizing information per photon.

Pioneering implementations span theoretical models and hardware: from joint-optimal optical-digital pipelines (Ma et al., 25 Mar 2026) and artificial synapses with in situ tunable photonic plasticity (Qin et al., 2016, Maity et al., 2024) to SPAD-based event streams processed by neuromorphic algorithms (Afshar et al., 2019) and photonic reservoirs with event-driven inputs (Skontranis et al., 17 May 2025). These approaches are unified by explicit photon-count and event statistics at all stages.

2. Mathematical and Physical Modeling

The statistical foundation of PANS departs from Gaussian-additive-noise approximations, instead modeling detection as a low-rate Poisson process (or its non-classical/quantum generalizations). For a detector channel $i$ with expected photon arrival $\lambda_i$ and background $b$, photon counts $y_i$ obey

$y_i \sim \mathrm{Poisson}(\lambda_i + b).$

In the single-photon detection regime, outputs are binary (click/no-click), with click probability $P(a_i=1|\lambda_i) = 1 - e^{-\lambda_i}$; the result is a Bernoulli random variable per channel (Ma et al., 25 Mar 2026).

End-to-end optimization for classification or inference proceeds by minimizing over both optical encoder parameters $W$ and digital network weights $\theta$: $$\min_{W,\,\theta}\; \mathbb{E}_{x,\,y \sim \mathrm{Poisson}(H_W(x) + b)}\big[L(f_\theta(y),\,\mathrm{label}(x))\big]$$ where $H_W(x)$ maps input to expected photon counts. Non-differentiability through the sampling layer is resolved by a straight-through estimator using the analytic damping factor,

$\lambda_i$0

implying that the backpropagated gradient $\lambda_i$1 gives an unbiased estimator for $\lambda_i$2 (Ma et al., 25 Mar 2026).

Device-level mathematical models for neuromorphic phototransistors or ferroelectric-gated graphene synapses similarly connect incident photon flux $\lambda_i$3 to internal states (trapped charge, polarization), with functional forms depending on device architecture (Qin et al., 2016, Maity et al., 2024). For event cameras, photon-to-event mappings are encoded as threshold-driven log-intensity increments producing digital spikes (Du et al., 2023).

3. Architectures and Hardware Realizations

Photon-Aware Neuromorphic Sensing has been instantiated across several physical and algorithmic platforms:

a. Joint-Optimized Optical-Electronic Pipelines

Optimal design leverages both the optical encoder—such as structured illumination patterns or diffractive photonic elements (passive or active)—and the neuromorphic digital back end (photon-activation layers, batch-normalized convolutional or MLP classifiers). System training uses initialization, slope annealing, photon-count regularization, and joint optimization under extreme photon constraints (Ma et al., 25 Mar 2026).

b. Neuromorphic Phototransistors

Graphene-nanotube hybrid and graphene-ferroelectric devices operate as optically driven synapses, supporting both short- and long-term plasticity. They leverage mechanisms such as photo-gating (charge trapping under optical excitation) and photovoltage-induced polarization changes, with nonvolatility and sub-ms switching attainable. Multilevel synaptic weights can be directly written optically with energy use as low as sub-fJ per event in upscaled architectures (Qin et al., 2016, Maity et al., 2024).

c. Single-Photon Avalanche Diode (SPAD) Event Cameras

SPAD arrays in direct time-of-flight mode produce timestamped single-photon events, which are then locally compressed and encoded into semantic event streams (First-AND, OOBU) prior to feature extraction, neuromorphic pooling, and classification. This structure yields orders-of-magnitude reductions in output data rate and substantial improvements in recognition latency and accuracy under extreme photon scarcity (Afshar et al., 2019).

d. Event-Based Neuromorphic Vision and Photonic Reservoirs

Recent systems integrate high-speed event cameras with photonic reservoirs such as time-delayed VCSEL extreme learning machines. Event-based pre-processing drastically reduces required memory and hardware compared to digital FCLs, while ultrafast time-multiplexing and binary spike coding enable high-throughput, low-power classification at the sensor edge (Skontranis et al., 17 May 2025).

Table: Representative PANS Hardware Paradigms

Approach	Sensing Principle	Notable Metrics
Joint opto-digital	Structured illumination	4.9 photons/inference, 73% acc.
Graphene hybrid synap.	Optical photo-gating	~250 nJ/event, τ ~5 ms
SPAD event camera	Avalanche + event coding	81× data reduction, 91.5% acc.
VCSEL reservoir	Photonic spike encoding	95.8% acc., 98.4% memory red.
Graphene-FE synap.	Photovoltaic gating	~0.4 fJ/event (scaled), ms timesc.
Event camera ODMR	Log-ADC + spike coding	13× speedup, >100× data red.

4. Experimental Validation and Performance

Empirical studies consistently demonstrate the superiority of photon-aware and neuromorphic approaches over conventional imaging under strong photon constraints. Key experimental findings include (Ma et al., 25 Mar 2026, Skontranis et al., 17 May 2025, Afshar et al., 2019, Du et al., 2023):

• In single-shot low-light classification (FashionMNIST, MNIST), PANS achieves 73–97% accuracy with 4.9–29 photons per inference—at least two orders of magnitude more photon-efficient than conventional imaging plus CNNs.
• SPAD-based event-driven classification achieves up to 91.5% per-recording accuracy with >80× data-rate reduction compared to frame-based processing.
• Graphene-based synaptic devices achieve multilevel analog conductance states, nonvolatile retention, sub-ms response dynamics, and compatibility with WDM or optical logic.
• Photonic neuromorphic processors for cytometry achieve accuracies above 95%, with 98–99.5% memory reduction, and low-latency event-driven operation.
• Event-based quantum sensing (NV diamond ODMR) yields a 13× gain in temporal resolution and >100× data reduction, while maintaining sensitivity and SNR (Du et al., 2023).

5. Theoretical Extensions and Adaptability

The PANS framework generalizes beyond classical Poisson photon detection, allowing substitution of more complex SPD statistics, such as those arising in quantum optics (squeezed states, entangled light) or with photon-number–resolving detectors (Ma et al., 25 Mar 2026). The stochastic modeling of detection events and their derivatives can be adapted by updating the event-generation probabilities and corresponding gradient estimators within the end-to-end optimization pipeline.

Hardware generality is similarly notable: PANS is applicable to both single-pixel (bucket) sensors, diffractive or photonic arrays, integrated neuromorphic pixels using 2D materials, and complex photonic reservoirs. Such adaptability extends its reach to new generations of quantum-aware, ultra-low-bandwidth, and in-memory/in-sensor computing platforms.

6. Applications and Implications

Photon-Aware Neuromorphic Sensing is poised to impact a spectrum of scientific and technological domains. Principal applications include:

• Ultra-low-dose fluorescence/multiphoton microscopy and flow cytometry, where minimized photon budgets are essential to reduce photodamage while retaining classification or quantification accuracy (Ma et al., 25 Mar 2026, Skontranis et al., 17 May 2025).
• Deep-space astronomical imaging, compact nebula or transient event detection, and speckle-contrast-based tissue perfusion under severe light-starvation (Ma et al., 25 Mar 2026).
• Event-driven 3D vision, high-speed robotics, covert surveillance, and resource-constrained edge AI, benefiting from both photon and hardware efficiency (Afshar et al., 2019, Skontranis et al., 17 May 2025).
• Widefield quantum sensing (e.g., NV-diamond magnetometry or nanoscale thermometry), exploiting the synergy of event-based data compression and high temporal precision (Du et al., 2023).
• All-optical and hybrid optoelectronic neuromorphic architectures, leveraging 2D material synapses and scalable photonic integration for brain-inspired computation (Qin et al., 2016, Maity et al., 2024).

A plausible implication is that continued progress in PANS device engineering—reducing energy per event, increasing scaling and reliability, and integrating event-driven learning at the sensor level—will facilitate the development of fully self-contained, computation-ready imagers and measurement systems operating at the theoretical minimum limit set by quantum detection statistics.

7. Outlook and Prospective Developments

PANS is closely aligned with broader trends in in-sensor and near-sensor computing, memristive and crossbar architectures, and the advent of hybrid analog-digital and spiking neural networks for efficient inference. Future directions include:

• Monolithic integration of optical synapses and event processing circuits for further reduction in latency and power.
• Realization of all-optical learning and adaptation directly at the detector level, exploiting recent advances in photonic neuromorphic device physics (Maity et al., 2024).
• Adaptive, feedback-driven event-coding in SPAD and event-camera arrays for closed-loop optimal photon allocation (Afshar et al., 2019).
• Application of PANS principles to quantum-enhanced sensing modalities and non-classical light sources (Ma et al., 25 Mar 2026, Du et al., 2023).

The PANS paradigm thus closes the methodological gap between the physics of measurement, device architecture, and machine learning-driven inference, defining a rigorous pathway for high-accuracy, low-resource, and quantum-aware sensing technologies.