Photonic Spiking TD3 for Continuous Control

• Photonic spiking TD3 is a reinforcement learning framework that fuses spiking neural networks with photonic substrates like DFB-SA lasers and MZI meshes for efficient continuous control.
• It leverages photonic hardware acceleration and hybrid neuromorphic design to achieve ultra-low latency and energy-efficient performance, as validated by high success rates in autonomous navigation.
• The architecture employs a twin-critic TD3 algorithm with a spiking actor and dense critic networks, enabling real-time, robust learning in robotics and autonomous systems.

Photonic Spiking Twin Delayed Deep Deterministic Policy Gradient (TD3) refers to a class of reinforcement learning (RL) architectures that integrate spiking neural networks (SNNs), photonic hardware accelerators, and the Twin Delayed Deep Deterministic Policy Gradient algorithm for continuous control. These systems leverage hardware-software codesign, mapping critical components of the RL pipeline onto photonic substrates—such as distributed feedback lasers with saturable absorbers (DFB-SA) or silicon Mach-Zehnder interferometer (MZI) meshes—achieving significant reductions in latency and energy consumption compared to conventional electronic implementations. The approach enables real-time, low-power control in robotics and autonomous navigation domains (Chen et al., 1 Feb 2026, Yu et al., 29 Nov 2025).

1. Reinforcement Learning Framework and TD3 Adaptation

The core algorithmic structure is based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, which augments standard actor-critic methods with dual critic networks to mitigate Q-value overestimation. In the photonic spiking TD3 framework:

• The Actor is implemented as a spiking neural network (SNN) and outputs continuous control actions, typically linear and angular velocities for robots.
• The Critic consists of two conventional dense neural networks estimating continuous Q-values, realizing the twin-critic structure of TD3.

Operationally, during a single RL cycle:

• The environment state $s$ (e.g., LiDAR readings plus navigation targets) is spike-encoded.
• The encoded state drives the photonic spiking Actor, which generates a spike train; this train is decoded into an action $a$ subject to exploration noise.
• The environment updates, returning $(r, s', d)$ tuples that populate a replay buffer.

Critic networks $Q_{\phi_1}, Q_{\phi_2}$ are updated by minimizing the mean-squared TD error:

$$L(\phi_j) = \mathbb{E}\left[\left(Q_{\phi_j}(s, a) - y\right)^2\right],\quad y = r + \gamma \min_{k=1,2} Q_{\phi_k'}(s', \pi_{\theta'}(s')+\epsilon)$$

where $\epsilon$ is clipped target policy smoothing noise and ${\phi_k'}, {\theta'}$ are target network parameters.

The Actor parameters are updated every two Critic steps by ascending the policy gradient:

$$\nabla_\theta J(\theta) = \mathbb{E}\left[\nabla_a Q_{\phi_1}(s, a)|_{a=\pi_\theta(s)} \nabla_\theta \pi_\theta(s)\right]$$

Policy and target networks are maintained with soft updates to ensure stable convergence (Chen et al., 1 Feb 2026, Yu et al., 29 Nov 2025).

2. Photonic and Hybrid Neuromorphic Hardware Realization

Two main photonic substrates are used:

DFB-SA Laser Array (Chen et al., 1 Feb 2026):

• The final nonlinear activation (LIF spiking layer) of the Actor network is mapped to a DFB-SA laser array, where each laser emulates a single Leaky Integrate-and-Fire neuron.
• The photonic membrane potential is driven by injected optical power; lasing threshold and gain/absorber regions reproduce spiking dynamics, including thresholding and refractory periods.
• The hidden layers, structured as positive-weight, bias-free matrices (e.g., 24–128–128–2), can be mapped directly to photonic linear computing cores.

MZI Meshes (Yu et al., 29 Nov 2025):

• A silicon-on-insulator 16×16 programmable MZI mesh implements linear matrix–vector multiplies for SNN layers.
• Each MZI performs 2×2 unitary operations, and the mesh approximates arbitrary real-valued weights via phase shifter voltages optimized with in-situ SPGD calibration.
• Photonic inputs are provided via modulated optical pulses; outputs are detected and used as electronic currents for subsequent LIF neuron integration.

In all cases, a hardware–software pipeline decouples linear and nonlinear operations for maximal hardware efficiency, with nonlinear spikes either realized photonicly (DFB-SA) or electronically (LIF after photonic MVM).

3. Spiking Neuron Dynamics and Photonic Device Modeling

The core neuron and device equations are:

Discrete-time LIF Dynamics:

$$U[t+1] = \lambda U[t] + \sum_i w_i x_i[t] - V_{\text{th}}, \quad S[t] = H(U[t] - V_{\text{th}})$$

where $U$ is the membrane potential, $\lambda$ the leak factor, $w_i$ synaptic weights, $x_i$ input spikes, $V_{\text{th}}$ threshold, $S[t] \in \{0,1\}$ the output spike, and $H$ the Heaviside function.

DFB-SA Laser Model (Yamada-style Rate Equations):

$$\begin{cases} \dfrac{dN_1}{dt} = \dfrac{I}{qV_1} - \dfrac{N_1}{\tau_{s}} - g_1(N_1 - N_{g1})S \ \dfrac{dN_2}{dt} = -\dfrac{N_2}{\tau_{s}} - g_2(N_2 - N_{g2})S \ \dfrac{dS}{dt} = [g_1(N_1 - N_{g1}) + g_2(N_2 - N_{g2})]S - \dfrac{S}{\tau_{\mathrm{ph}}} + S_{\mathrm{ext}} + B_{\mathrm{sp}} \end{cases}$$

where $N_1, N_2$ are carrier densities (gain/absorber), $S$ photon density, $g_1, g_2$ gain coefficients, $\tau_s, \tau_{\mathrm{ph}}$ lifetimes, $S_{\mathrm{ext}}$ injected photons.

Photonic Cost Models:

• DFB-SA array total electrical power: $P_{\text{DFB}} = 4.10$ W.
• Inference latency: $\tau_{\text{inf}} = 191.20$ ps (spike frequency $f_{\text{spike}} = 5.23$ GHz).
• Energy per inference: $E_{\text{inf}} \approx 0.78$ nJ/inf.

On MZI meshes, photonic-limited per-layer latency is 120 ps; energy efficiency is 1.39 TOPS/W (Yu et al., 29 Nov 2025).

4. Experimental Validation and Performance Benchmarks

Autonomous Navigation with DFB-SA Array (Chen et al., 1 Feb 2026):

• Gazebo simulation, dynamic obstacle avoidance: Average reward $58.22 \pm 17.29$, success rate $80\% \pm 8.3\%$.
• Hardware–software co-inference (DFB-SA LIF layer only): For 5,888 (no obstacle) and 11,904 (with obstacles) test examples, error rates were 0.051% and 0.059%, respectively.
• Inference latency: $191.20$ ps/inf, compared to $2,083$ ps/inf on Eyeriss.
• Energy: $0.78$ nJ/inf, compared to $11.98$ μJ/inf on PopSAN.

Robotic Continuous Control with MZI Mesh (Yu et al., 29 Nov 2025):

• Pendulum-v1: both pure software and photonic co-inference converge at –146 reward in $7.4 \times 10^4$ steps.
• HalfCheetah-v2: Photonic co-inference converges to $5831 \pm 495$ reward in $5.75 \times 10^5$ steps (23.3% faster than $7.5 \times 10^5$ for software), with action deviation below 2.2%.
• Cosine similarity between target and hardware-implemented weights $> 0.81$ after SPGD calibration.

Validation of Error Channels (Chen et al., 1 Feb 2026):

• Yamada-model simulation of DFB-SA error channels demonstrates perfect agreement between simulated responses and software targets, confirming that device-level dynamics underlie observed activations.

Experimental Setup Highlights:

• DFB-SA: Tunable laser, MZM, circulator, photodetector, oscilloscope; 27.9 mA gain current, 0.76 V SA bias, $I_{\text{th}} = 18$ mA, SMSR $>45$ dB.

5. Pipeline Integration and Algorithmic Workflow

The full hardware–software inference loop comprises:

1. Sensory state quantization and spike encoding.
2. Driving photonic linear algebra units (DFB-SA or MZI mesh) with optical pulses.
3. Photonic computation of weight matrices.
4. Detection/conversion of optical outputs to membrane-integrated currents or to further SNN layers.
5. Spiking activity decoded to produce real-valued (continuous) actions.
6. Action issued to robotic environment (Gazebo-ROS, MuJoCo).
7. Critic evaluation and storage for replay-based TD3 training.

Spike-rate encoding is frequently used to translate continuous state variables into spike trains over several time steps (typically $T=1$ for DFB-SA, $T=1$ or $T=4$ for MZI systems).

6. Scalability, Limitations, and Integration Prospects

Ablation studies with varying SNN hidden-layer sizes and time steps (100×100, 128×128, 256×256, 512×512; $T=1,2,4$) indicate that 128×128 with $T=1$ achieves optimal convergence and energy-latency tradeoff (Chen et al., 1 Feb 2026). The main photonic mesh implementations are currently limited by fabrication constraints (e.g., 16×16 for MZI); simulations show that larger meshes further improve performance but introduce calibration complexity related to phase shifter count and thermal stabilization (Yu et al., 29 Nov 2025).

The positive-weight, bias-free constraints of photonic Actors allow immediate mapping to existing photonic matrix–vector multiply cores, with on-chip DFB-SA or electrical LIF implementations providing nonlinearities. End-to-end photonic critic integration remains a challenge.

7. Implications for Autonomous Robotics and Photonic Computing

Photonic Spiking TD3 architectures achieve ultra-low inference latency and energy consumption, supporting real-time, on-board learning and decision-making in robotic platforms. The demonstrated error rates, reward convergence, and system robustness validate the viability of large-scale photonic neuromorphic RL. Integration of both linear and nonlinear SNN components on photonic chips, and mapping of SNN/ANN frameworks (with appropriate constraints) onto these substrates, suggests a pathway to scalable, fully photonic RL accelerators suitable for embedded and edge-deployed intelligent systems, offering performance beyond that of conventional von Neumann electronic hardware (Chen et al., 1 Feb 2026, Yu et al., 29 Nov 2025).