Spiking Deep Belief Networks

• Spiking DBNs are deep probabilistic generative models that combine layered architectures with spiking neurons to approximate sigmoid sampling using stochastic integrate-and-fire dynamics.
• The approach leverages neuromorphic hardware such as IBM TrueNorth to perform fast, energy-efficient Gibbs sampling using integer operations and locally generated noise.
• This architecture supports robust classification and generative modeling on tasks like MNIST digit recognition, with careful parameter tuning balancing accuracy and sampling quality.

Spiking Deep Belief Networks (Spiking DBNs) are a class of probabilistic generative models that combine deep layered architectures such as Deep Belief Networks (DBNs) or Restricted Boltzmann Machines (RBMs) with spiking digital neuron substrates. These systems are designed to leverage the event-driven, low-power, and massively parallel properties of neuromorphic hardware, in particular digital implementations such as IBM’s TrueNorth processor. In Spiking DBNs, the key nonlinear operation—the sampling of binary units according to a logistic-sigmoid conditional probability—is approximated by the firing statistics of stochastic integrate-and-fire (I&F) neurons subjected to noise in both leak and threshold. This approach enables efficient implementation of @@@@1@@@@ (MCMC) inference, notably Gibbs sampling, using integer operations and simple random number generation rather than floating-point exponentiation or lookup tables typical in von Neumann architectures (Das et al., 2015).

1. Spiking Neuron Model and Sigmoid Approximation

In the TrueNorth realization, each RBM or DBN unit is mapped to a digital integrate-and-fire neuron. The neuron's membrane potential $V_j(t)$ is updated at each clock step $t$ as follows:

$$\begin{aligned} V_j(t) &= V_j(t-1) + \sum_{i=0}^{N-1} x_i(t)s_{ij}, \ V_j(t) &\leftarrow V_j(t) - \lambda_j, \ \text{if } V_j(t) \geq \alpha_j & : \text{emit a spike and set } V_j(t) = R_j, \end{aligned}$$

where $x_i(t) \in \{0,1\}$ denotes binary spikes, $s_{ij}$ are integer weights, $\lambda_j$ is a stochastic leak, $\alpha_j$ is a stochastic threshold, and $R_j$ is the reset value.

To realize the binary sampling step

$$P(x_j=1\,|\,\{x_i\}_{i\neq j}) = \sigma(\sum_i w_{ij} x_i + b_j),$$

multiple sub-intervals ($T_w$ steps) are simulated. In each, the membrane potential is updated with random leak and threshold noise. If the membrane potential exceeds a randomized threshold in any sub-interval, the unit's binary state is set to one.

The resultant sampling probability across $T_w$ steps closely approximates the desired sigmoid,

$$P(x=1|v) \approx 1 - [1 - P_{\text{step}}(v)]^{T_w} \approx \sigma(v/s),$$

where $P_{\text{step}}(v)$ is the spiking probability in one sub-interval and $s$ is a hardware-dependent scale factor (Das et al., 2015).

2. Network Architecture and Hardware Mapping

The demonstrated architecture uses RBMs with the following topologies:

• Classification: $784$ visible units (grayscale pixels), $500$ hidden units, and $10$ label units for MNIST digit classification.
• Generative tests: $3$ visible and $2$ hidden units for toy-model validation.

In Spiking DBNs, each logical unit corresponds to a physical digital neuron. Weights and biases learned offline are quantized and scaled using an integer factor ($\mathrm{scale}>1$), affecting the dynamic range and sigmoid sharpness:

$$P_{\rm scaled}(v) = \frac{1}{1 + \exp(-v/\mathrm{scale})}.$$

TrueNorth's infrastructure offers 4096 cores, each with 256 neurons, enabling full-mapping of even substantial RBMs. A global "leak neuron" broadcasts stochastic leak values, and per-neuron pseudo-random number generators produce threshold noise (controlled by $TM$ bits of entropy) for local sampling.

3. Gibbs Sampling Algorithm and Training Workflow

Gibbs sampling in this context alternates between updating visible and hidden states. For classification, only a single visible-to-hidden update (one-step inference) is typically performed; for generative modeling, many iterations (up to $10^5$) approximate the model's equilibrium distribution.

The training procedure is as follows:

• Offline: Standard Contrastive Divergence (CD) on CPU/GPU platforms, yielding learned weights and biases.
• Online: Fixed parameters are uploaded to the chip.
• No on-chip learning is implemented.

MCMC sampling proceeds through parallel update of all units using the hardware-based digital Gibbs sampler for $T_w$ steps per unit per sweep. No burn-in is included for the classification setting; for generative evaluation, results are reported after a large fixed number of iterations.

4. Performance Characterization

4.1. Classification Accuracy

On the MNIST task (1000 test digits, training on 5000 digits), multiple sampler parameterizations were evaluated. The following table summarizes the main digital neuron settings, which collectively determine sigmoid approximation and sampling dynamics:

Index	$(T_w, Vt, TM, \mathrm{leak})$	Scale
P1	(1, -130, 8, 0)	50
P2	(1, -80, 8, 102)	50
P3	(1, -20, 8, 200)	75
P4	(1, -100, 9, 300)	120
P5	(16, 50, 9, 15)	30
P6	(16, 100, 10, 30)	50
P7	(16, 633, 8, 90)	100

On noise-free test data, all configurations achieved classification accuracy within ≈1% of the ideal (CPU-based) logistic-sigmoid RBM classifier. Under added noise (salt and salt-and-pepper), higher scale and nonzero leak (P4, P7) led to significantly better robustness. Low-resolution configurations (small scale, zero leak) degraded rapidly as input noise increased.

4.2. Generative Model Quality

For a toy RBM with known Boltzmann distribution, Kullback-Leibler (KL) divergence after $10^5$ sampling iterations reflected the match of the sampler to the true model:

Comparison	Trial 1	Trial 2	Trial 3
exact vs. ideal	$6.2\times10^{-5}$	$6.1\times10^{-5}$	$5.7\times10^{-5}$
exact vs. digital (no leak)	0.0218	0.1090	0.0259
exact vs. digital (leak)	0.0091	0.0330	0.0083

Zero-leak samplers produced KL divergences far above the ideal; nonzero-leak improved sampling quality but remained two orders of magnitude worse than the floating-point ideal sampler.

4.3. Hardware Metrics

• Latency: One Gibbs sampling pass with $T_w = 1$ completes in 3 cycles (1 update, 2 synchronization).
• Power: While no absolute numbers are cited, prior TrueNorth benchmarks suggest tens of milliwatts for 1 million neurons, yielding sub-microjoule energy per sample for these network sizes.
• Area: Full sampling operation is performed entirely on-chip (4096 cores × 256 neurons), with all routing and memory integrated, requiring no external SRAM.

In comparison to CPU, GPU, or FPGA sigmoid units (which require multiplications, exponentiations, or LUTs), this approach uses exclusively integer addition, comparison, and simple PRNGs, yielding orders of magnitude lower energy per Gibbs sample. However, careful tuning of parameters ($T_w, Vt, TM$, leak, scale) is needed due to quantization and limited dynamic range.

5. Discussion, Limitations, and Prospects

The feasibility of deploying Gibbs sampling for RBMs/DBNs in a digital neuromorphic setting with local, stochastic spiking neurons is established. For discriminative tasks, even the minimal mechanism ($T_w=1$, zero leak) incurs negligible accuracy loss. In contrast, successful generative modeling (e.g., faithful Boltzmann sampling) requires nonzero stochastic leak for proper mixing and lower KL divergence.

Strengths of this approach include ultra-low-latency and -power sampling thanks to the event-driven, parallel architecture. The trade-offs involve the absence of on-chip learning (weights must be pre-trained and quantized externally), and the necessity of parameter scaling for the hardware’s fixed-point arithmetic.

Planned future directions include incorporation of on-chip CD-style learning, extension to deeper DBNs, dynamic noise adaptation, and real-time applications in ultra-low-power domains such as IoT and brain–computer interfaces. The demonstrated methods provide foundational infrastructure for further neuromorphic algorithm–hardware codesign (Das et al., 2015).