Training slow silicon neurons to control extremely fast robots with spiking reinforcement learning

Abstract: Air hockey demands split-second decisions at high puck velocities, a challenge we address with a compact network of spiking neurons running on a mixed-signal analog/digital neuromorphic processor. By co-designing hardware and learning algorithms, we train the system to achieve successful puck interactions through reinforcement learning in a remarkably small number of trials. The network leverages fixed random connectivity to capture the task's temporal structure and adopts a local e-prop learning rule in the readout layer to exploit event-driven activity for fast and efficient learning. The result is real-time learning with a setup comprising a computer and the neuromorphic chip in-the-loop, enabling practical training of spiking neural networks for robotic autonomous systems. This work bridges neuroscience-inspired hardware with real-world robotic control, showing that brain-inspired approaches can tackle fast-paced interaction tasks while supporting always-on learning in intelligent machines.

• The paper demonstrates a spiking neural network (SNN) achieving 96–98% success rates in controlling a robotic arm for high-speed interception using 1020 silicon neurons.
• The method employs a reward-modulated local e-prop learning rule and fixed random recurrent connectivity in the hidden layer, with plug from population codes via spike generators.
• The system shows robustness across multiple conditions with minimal neural representation, suggesting scalability and efficiency in neuromorphic hardware applications.

Summary of the work

This paper demonstrates reinforcement learning (RL) for a fast robotic interception task — air hockey — using a compact spiking neural network (SNN) executed on the DYNAP-SE mixed-signal analog/digital neuromorphic processor. The system operates in a hardware-in-the-loop configuration: a MuJoCo simulation provides the environment, while policy inference runs on 1020 silicon AdEx-LIF neurons at every 50 Hz control step. Learning is restricted to the readout layer and uses a local e-prop-based rule that approximates backpropagation through time using only pre- and post-synaptic activity and a scalar reward. The hidden layer consists of fixed random recurrent connectivity (a reservoir), so all plasticity is confined to a small set of readout weights updated every two trials.

The central empirical claim is that this architecture achieves 96–98% success rates over 2000 episodes under randomized puck initial conditions, with puck speeds up to 1.5 m/s, using an order of magnitude fewer neurons than prior simulation-based approaches to the same task (1020 vs. 10,000). The paper positions itself as scaling neuromorphic RL beyond toy benchmarks such as Atari Pong toward physically grounded manipulation with continuous state spaces and millisecond-precision timing constraints.

System architecture and learning rule

The pipeline comprises three stages. First, a host CPU encodes the 6D continuous state — puck position and velocity $(x_p, y_p, v_x, v_y)$ plus end-effector position $(x_{ee}, y_{ee})$ — into population codes, with each variable represented by a Gaussian activity profile across 10 neurons, yielding 60 FPGA-based spike generators. Second, these spike trains drive 1020 silicon AdEx-LIF neurons with fixed random feedforward synapses; the analog neuron dynamics process each observation for 20 ms before readout. Third, two linear readout units compute action probabilities via softmax over filtered hidden-layer activity, selecting between two motion primitives (a "home" target at $x = 0.70$ m and a "forward" intercept target at $x = 1.50$ m) executed as open-loop spline trajectories through an inverse kinematics solver.

The plasticity is confined to the hidden-to-readout weights, updated according to:

$$\Delta w_{ik} = -\alpha \sum_t r^t \sum_{t' \leq t} \gamma^{t-t'} (\pi_{k}^{t'} - \mathbb{1}_{a^{t'}=k}) \bar{s}_{i}^{t'}$$

where $\bar{s}_i^{t'}$ is the filtered presynaptic trace, $r^t$ the reward, $\gamma$ the discount factor, and $(\pi_k - \mathbb{1}_{a=k})$ the policy-gradient advantage term. This is a fully local, online rule requiring no backpropagation machinery on-chip.

The reward function combines three components: a large terminal bonus (+20) for driving the puck past a threshold position with forward velocity, a dense shaping term proportional to forward puck speed when the end-effector is within contact distance ($d \leq 0.08$ m), and a small per-step penalty ($(x_{ee}, y_{ee})$0) encouraging fast episode completion. Success is defined as the first condition — propelling the puck forward past $(x_{ee}, y_{ee})$1 m with $(x_{ee}, y_{ee})$2 m/s — which the authors should note is a task-level criterion rather than merely physical contact.

Key results

The evaluation covers two axes. On task-level generalization, four conditions of increasing difficulty were tested across 10 independent reservoir seeds (i.e., independent random connectivity instantiations), with results reported as mean and interquartile spread:

Condition	Asymptotic success	Episodes to convergence
Stationary puck	100%	~200
Constant-speed launch	100%	~1000
Speed variability [1.0, 1.5] m/s	>96%	~1500
Randomized position + speed	>98%	~1300

A notable finding is that the hardest condition — randomizing both initial position within a 0.10 m window and speed — yields the highest asymptotic performance. The authors attribute this to broader state distributions preventing readout overfitting, an effect consistent with reservoir-computing literature but worth verifying explicitly.

On encoding-range scalability, fixing the network size while widening the puck velocity range produces graceful degradation: [0.7, 0.9] m/s reaches >97% in ~150 episodes; [0.7, 1.2] m/s requires ~700 episodes; [0.7, 1.5] m/s drops asymptotic success from 97% to 93%; and the widest range ([0.5, 2.0] m/s) falls to 86%. This quantifies the finite representational resolution of a fixed-size silicon reservoir and suggests that capacity, not algorithm, becomes the binding constraint as input ranges widen.

Qualitatively, the learned policy consolidates from near-random exploration into a deterministic strategy of early commitment to the interception trajectory followed by precisely timed corrective switches between primitives, with markedly reduced timing variance post-training.

Relation to prior work

The paper extends Blakowski et al.'s DYNAP-SE implementation of awake/dreaming RL from Atari Pong, adapting it from a 4D discrete pixel space to a 6D continuous metric space and from instantaneous actions to composed motion primitives. Compared to Capone et al.'s simulation-only framework, the hardware realization here uses 10× fewer neurons than Ambrosini et al.'s simulation-based air hockey result (1020 vs. 10,000). Relative to DeWolf et al.'s Loihi-based control of a simulated 7-DOF arm, which used hand-engineered neural controllers rather than learned policies, this work demonstrates online RL policy acquisition in spiking hardware. However, the comparison table in the paper omits several directly relevant neuromorphic robot-control demonstrations, and the novelty claim rests primarily on domain scaling rather than algorithmic innovation — a point reviewers correctly flagged.

Limitations and open questions

Several limitations constrain the strength of the claims, and the paper would benefit from addressing them directly:

• Simulation only. Despite the title's reference to controlling robots, all experiments run in MuJoCo; there are no physical robot trials, no sim-to-real transfer experiments, and no discussion of sensor noise or mechanical perturbations. The claim of "real-world robotic control" is therefore overstated relative to the evidence.
• No energy or latency measurements. Energy efficiency motivates the entire approach, yet no power figures for chip-plus-host operation, no latency breakdown (encoding, FPGA transfer, 20 ms neuron settling, decoding), and no comparison against GPU or CPU baselines are provided. Since encoding, eligibility-trace computation, and weight updates appear to run off-chip on the host, the effective energy advantage may be substantially smaller than the chip's intrinsic efficiency suggests.
• Missing baselines. No comparison against PPO, SAC, TD3, or even a lightweight ANN or hand-tuned heuristic on the same task, making sample efficiency and performance claims difficult to contextualize.
• Limited DOF utilization. The task is planar; the anthropomorphic arm's additional degrees of freedom execute fixed open-loop splines and are not themselves subject to learning, so the effective control problem is closer to 2D than the hardware suggests.
• Off-chip learning. Readout updates occur on the host every two trials; fully on-chip plasticity remains future work, as does deployment on platforms such as iCub with native event-camera input.

An open technical question the paper leaves unresolved is whether the observed degradation at wide encoding ranges reflects fundamental limits of fixed-size reservoirs or merely suboptimal population-code allocation, and whether multi-core DYNAP-SE configurations recover full performance.

Conclusion

The paper provides a credible demonstration that local, reward-modulated plasticity in a small fixed-reservoir SNN can acquire temporally precise interception policies for a fast, continuously valued robotic task, achieving 96–98% success with 1020 silicon neurons and robustness across 10 random connectivity seeds. Its principal contribution is architectural and empirical scaling rather than algorithmic novelty. The absence of physical-robot validation, quantitative energy/latency measurements, and conventional RL baselines means the central thesis — that neuromorphic hardware confers practical advantages for autonomous robotic learning — remains supported by plausibility rather than measurement, and closing that evidentiary gap is the most pressing follow-up this work identifies.