Nonnegative Spiking RNN Autoencoder

• Nonnegative Spiking RNN Autoencoder is a spike-based, probabilistic architecture that uses nonnegative, row-normalized weights and saturating activations to enable unsupervised feature extraction.
• It employs a feed-forward structure with event-driven spiking dynamics and NMF-inspired multiplicative updates to minimize mean square reconstruction error.
• Experimental evaluations on datasets like MNIST, CIFAR-10, and UCI benchmarks validate its robustness, scalability, and potential for neuromorphic hardware implementations.

The Nonnegative Spiking Random Neural Network (RNN) Autoencoder is a neural architecture that integrates a feed-forward, spike-based computational framework with strict nonnegativity and probabilistic constraints on weights, using Nonnegative Matrix Factorization (NMF)-inspired learning algorithms. It is designed to perform efficient, distributed representation learning, supporting both shallow and deep (multi-layer) autoencoder structures, and amenable to implementation in event-driven, neuromorphic hardware. This model was introduced to address both the computational features of biologically plausible spiking networks and the representational constraints required by nonnegativity, providing a platform for unsupervised feature extraction and reconstruction on a range of real-world datasets (Yin et al., 2016).

1. Spiking Random Neural Network Foundations

The spiking Random Neural Network (RNN) model defines neurons by integer-valued "potential" variables, $k_h(t) \ge 0$, where each neuron $h$ has a steady-state excitation probability $q_h = \Pr\{k_h>0\}$. In the general RNN, neurons communicate through excitatory and inhibitory spikes, each firing at a rate $r_v$. The steady-state activity satisfies a nonlinear, saturating fixed-point equation: $$q_h = \min\!\left( \frac{\lambda_h^+ + \sum_{v=1}^N q_v\,r_v\,p^+_{vh}}{r_h + \lambda_h^- + \sum_{v=1}^N q_v\,r_v\,p^-_{vh}},\,1 \right)$$ with $\lambda_h^+$ and $\lambda_h^-$ denoting external excitatory and inhibitory spike rates.

The Nonnegative Spiking RNN Autoencoder adopts a simplified, feed-forward (quasi-linear) form: all connections are excitatory, weights are nonnegative, inputs are modeled as constant external Poisson spike rates $x_v \ge 0$, and spiking is unidirectional (layer-to-layer). The effective weights $w_{v,h} = p^+_{v,h}\,r_v$ are subject to normalization $\sum_h w_{v,h} \le 1$, and the excitation probabilities become: $h$0 guaranteeing all activations remain within probabilistic bounds.

2. Architectural Structure and Constraints

The autoencoder comprises an input layer, one or more hidden (encoding) layers, and output (decoding) layers. All inter-layer weights $h$1 are nonnegative and row-normalized so that for each neuron, the sum of outgoing weights does not exceed 1: $h$2 This enforces a probabilistic interpretation, with each spike leaving neuron $h$3 routed to $h$4 with probability $h$5 or lost otherwise. The network processes input $h$6 via layer-wise deterministic saturating linear transforms: $h$7 Where $h$8 is applied elementwise and all matrix dimensions are as per layer sizes.

The model generalizes to depth $h$9 by repeating the encoding and decoding stages: $q_h = \Pr\{k_h>0\}$0 All weights in all layers obey nonnegativity and row-sum constraints, ensuring probabilistic, spike-based propagation through the hierarchy.

3. NMF-inspired Learning Algorithm

Training minimizes the mean square reconstruction error between inputs $q_h = \Pr\{k_h>0\}$1 and outputs $q_h = \Pr\{k_h>0\}$2: $q_h = \Pr\{k_h>0\}$3 subject to all RNN nonnegativity and normalization constraints. The approach leverages NMF-style multiplicative update rules (elementwise): $q_h = \Pr\{k_h>0\}$4

$q_h = \Pr\{k_h>0\}$5

with $q_h = \Pr\{k_h>0\}$6 denoting elementwise multiplication and division handling zeros via an "eps" stabilizer.

After each update, normalization ensures row sums do not exceed one, both for $q_h = \Pr\{k_h>0\}$7 and $q_h = \Pr\{k_h>0\}$8. Additional global normalization (scaling by the maximal activation) prevents activation saturation. The multilayer architecture extends these updates layerwise using activations from $q_h = \Pr\{k_h>0\}$9 and $r_v$0 in the corresponding formulae, always maintaining the nonnegativity and probabilistic normalization after every iteration.

4. Experimental Evaluation

Empirical assessment used several image and tabular datasets:

• MNIST: $r_v$1 training and $r_v$2 test grayscale images ($r_v$3; values in $r_v$4).
• Yale Face: $r_v$5 faces, resized to $r_v$6.
• CIFAR-10: $r_v$7 training/$r_v$8 test RGB images, $r_v$9 ($$q_h = \min\!\left( \frac{\lambda_h^+ + \sum_{v=1}^N q_v\,r_v\,p^+_{vh}}{r_h + \lambda_h^- + \sum_{v=1}^N q_v\,r_v\,p^-_{vh}},\,1 \right)$$0 features), normalized to $$q_h = \min\!\left( \frac{\lambda_h^+ + \sum_{v=1}^N q_v\,r_v\,p^+_{vh}}{r_h + \lambda_h^- + \sum_{v=1}^N q_v\,r_v\,p^-_{vh}},\,1 \right)$$1.
• 16 UCI datasets: attributes normalized to $$q_h = \min\!\left( \frac{\lambda_h^+ + \sum_{v=1}^N q_v\,r_v\,p^+_{vh}}{r_h + \lambda_h^- + \sum_{v=1}^N q_v\,r_v\,p^-_{vh}},\,1 \right)$$2.

Architectures included shallow autoencoders (e.g., $$q_h = \min\!\left( \frac{\lambda_h^+ + \sum_{v=1}^N q_v\,r_v\,p^+_{vh}}{r_h + \lambda_h^- + \sum_{v=1}^N q_v\,r_v\,p^-_{vh}},\,1 \right)$$3, $$q_h = \min\!\left( \frac{\lambda_h^+ + \sum_{v=1}^N q_v\,r_v\,p^+_{vh}}{r_h + \lambda_h^- + \sum_{v=1}^N q_v\,r_v\,p^-_{vh}},\,1 \right)$$4) and deep (e.g., $$q_h = \min\!\left( \frac{\lambda_h^+ + \sum_{v=1}^N q_v\,r_v\,p^+_{vh}}{r_h + \lambda_h^- + \sum_{v=1}^N q_v\,r_v\,p^-_{vh}},\,1 \right)$$5). Training used mini-batch SGD, with batch sizes adapted per dataset (MNIST, CIFAR: 100; Yale: 5; UCI: 50), weights initialized to obey RNN constraints, and optimization either for a set epoch count or until mean square error (MSE) plateaued.

Performance was consistently evaluated by mean square reconstruction error: $$q_h = \min\!\left( \frac{\lambda_h^+ + \sum_{v=1}^N q_v\,r_v\,p^+_{vh}}{r_h + \lambda_h^- + \sum_{v=1}^N q_v\,r_v\,p^-_{vh}},\,1 \right)$$6

Key quantitative results:

Dataset	Architecture	MSE (Shallow)	MSE (Multi-layer)
MNIST	$$q_h = \min\!\left( \frac{\lambda_h^+ + \sum_{v=1}^N q_v\,r_v\,p^+_{vh}}{r_h + \lambda_h^- + \sum_{v=1}^N q_v\,r_v\,p^-_{vh}},\,1 \right)$$7	$$q_h = \min\!\left( \frac{\lambda_h^+ + \sum_{v=1}^N q_v\,r_v\,p^+_{vh}}{r_h + \lambda_h^- + \sum_{v=1}^N q_v\,r_v\,p^-_{vh}},\,1 \right)$$8	$$q_h = \min\!\left( \frac{\lambda_h^+ + \sum_{v=1}^N q_v\,r_v\,p^+_{vh}}{r_h + \lambda_h^- + \sum_{v=1}^N q_v\,r_v\,p^-_{vh}},\,1 \right)$$9
Yale faces	$\lambda_h^+$0	$\lambda_h^+$1	more stable for shallow
CIFAR-10	$\lambda_h^+$2	$\lambda_h^+$3	$\lambda_h^+$4
UCI datasets	various	steadily decreasing in all 16 cases	—

This demonstrates the model's broad applicability and stability across widely varying domains and input dimensionalities.

5. Stochastic Spiking Simulation and Hardware Implications

A numerical event-driven spiking simulation was performed: external Poisson spikes drive input neurons at $\lambda_h^+$5, each firing at rate 1, passing spikes according to $\lambda_h^+$6. At intervals, the potential $\lambda_h^+$7 of neuron $\lambda_h^+$8 is measured, estimating average excitation: $\lambda_h^+$9 yielding an excitation probability estimate

$\lambda_h^-$0

After $\lambda_h^-$1 events, the empirical $\lambda_h^-$2-values in all layers closely match the numerically calculated probabilities from the deterministic feed-forward equations, aligning the event-driven stochastic network with the idealized nonnegative RNN autoencoder. This correspondence supports the implementation of the architecture in massively parallel, asynchronous spiking neuromorphic systems.

6. Model Trade-offs and Design Considerations

Multiple trade-offs shape both design and application:

• Sparsity vs. Accuracy: The row-sum constraint $\lambda_h^-$3 imparts a sparsity pressure, but excessive normalization can limit representational power. Practical performance is balanced by tuning the hidden layer size $\lambda_h^-$4.
• Computational Speed vs. Distributability: Batch NMF-style updates facilitate rapid convergence in conventional hardware, while true event-driven spiking implementation sacrifices wall-clock speed for asynchronous, distributed operation with potential power efficiency.
• Depth vs. Stability: Shallow architectures exhibit slightly more stable convergence; deeper multi-layer stacks achieve marginally lower MSE at the expense of more complex normalization and propagation.

A plausible implication is that different use cases may prioritize either the rigorous distributed spiking implementation (for neuromorphic chips) or fast NMF-based training (for standard hardware), enabling unique deployment flexibility.

7. Significance and Broader Impact

The Nonnegative Spiking RNN Autoencoder establishes a framework where biologically inspired spike-based processing, strict nonnegativity, and proven NMF-style learning synergize, enabling unsupervised feature learning on diverse data. Its mathematical formulation guarantees compatibility with probabilistic spiking dynamics and supports hardware realizability in distributed, event-driven platforms. Its demonstrated applicability to standard benchmarks (MNIST, Yale, CIFAR-10) and UCI datasets evidences robustness and scalability. The architecture provides a concrete pathway for integrating low-power spike processing with modern unsupervised learning, with trade-offs enabling adaptation to diverse computational environments (Yin et al., 2016).