---
title: 'AdaBLDM: Adaptive Deep Belief Networks'
url: https://www.emergentmind.com/topics/adabldm-algorithm
type: topic
---

# AdaBLDM: Adaptive Deep Belief Networks

AdaBLDM (Adaptive Learning Method of Deep Belief Network by Layer Generation) is an adaptive deep architecture learning algorithm that automatically determines both the width (number of hidden units) and depth (number of layers) of Deep Belief Networks (DBNs) during training. Introduced by Kamada & Ichimura, AdaBLDM augments the standard DBN framework by equipping layerwise-trained Restricted Boltzmann Machines (RBMs) with structural learning mechanisms, including neuron generation, neuron annihilation, structural sparsity via forgetting, and global layer-generation criteria. These features enable AdaBLDM to produce a compact, sparse, and interpretable DBN optimized for the complexity of a given dataset, attaining state-of-the-art accuracy on image classification benchmarks [1807.03486].

## 1. Problem Formulation and Motivation

Standard DBNs require manual selection of architecture, fixing both the size of each RBM layer and the total number of layers before learning. This static design leads to well-known trade-offs:
- **Underfitting**: Too small a model cannot capture data regularities.
- **Overfitting and inefficiency**: Oversized networks are costly to train, prone to overfitting, and difficult to interpret.

AdaBLDM addresses this by introducing an adaptive mechanism that discovers:
- The optimal number of hidden units (neurons) in each RBM,
- A sparse connectivity structure,
- The optimal number of RBM layers needed for hierarchical feature extraction.

This reduces the need for manual architecture tuning and controls both over- and underfitting dynamically during learning [1807.03486].

## 2. Algorithmic Workflow and Pseudocode

AdaBLDM proceeds in two nested, mutually adaptive loops:
- **Layerwise training of each RBM (Contrastive Divergence, CD-1) with adaptive width and sparsification,**
- **Evaluation of criteria for growing a new layer atop the current DBN.**

A high-level pseudocode summary:

```python
# Inputs: dataset V, learning rate η, initial hidden h₀
# Neuron gen/annihilation thresholds: θ_G, θ_A
# Layer-gen thresholds: θ_L1, θ_L2; forgetting coeffs: ε₁, ε₂, ε₃
# Scaling factors: α_c, α_W, α_WD, α_E
k = 1
Initialize RBM_1 with h₀ hidden units
while True:
    Pretrain RBM_k with adaptive neuron generation/annihilation and SLF
    for each mini-batch:
        CD-1 update of weights/biases
        If (α_c * ||∇c_j||) * (α_W * ||∇W_[:,j]||) > θ_G: generate neuron j
        If mean activation of h_j < θ_A: annihilate neuron j
        Apply forgetting penalties to weight update
    After epoch: compute layer energy E^k and variance WD^k
    Compute G_WD = Σ_{l=1}^k α_WD*WD^l; G_E = Σ_{l=1}^k α_E*E^l
    If G_WD > θ_L1 and G_E > θ_L2:
        k += 1
        Initialize RBM_{k} by inheriting parameters from RBM_{k-1}
    else:
        break
# Optional fine-tuning (e.g. classifier layer)
Output: final DBN with discovered architecture
```

[1807.03486]

## 3. Mathematical Formulation

### 3.1. RBM Layer Energy Model

The RBM layer models a joint visible-hidden distribution:
\[
E(v,h) = -\sum_{i} b_i v_i - \sum_{j} c_j h_j - \sum_{i,j} v_i W_{ij} h_j
\]
\[
p(v,h) = \frac{1}{Z} \exp(-E(v,h)), \qquad Z=\sum_{v,h} \exp(-E(v,h))
\]

### 3.2. Contrastive Divergence (CD-1) Update

For weights and biases:
\[
\Delta W_{ij} = \eta \left( \langle v_i h_j \rangle_{data} - \langle v_i h_j \rangle_{model} \right)
\]
\[
\Delta b_{i} = \eta \left( \langle v_i \rangle_{data} - \langle v_i \rangle_{model} \right)
\]
\[
\Delta c_{j} = \eta \left( \langle h_j \rangle_{data} - \langle h_j \rangle_{model} \right)
\]

### 3.3. Adaptive Neuron Generation/Annihilation

- **Generation**: If for neuron $j$,
  \[
  (\alpha_{c}\,||d c_j||)\;\cdot\;(\alpha_{W}||d W_{:,j}||) > \theta_{G}
  \]
  then neuron $j$ is split/generated.

- **Annihilation**: If
  \[
  \frac{1}{N} \sum_{n=1}^N p(h_j=1|v^{(n)}) < \theta_{A}
  \]
  neuron $j$ is pruned.

### 3.4. Structural Learning with Forgetting (SLF)

Three penalty terms encourage sparsity and binary activations:
- **L1 forgetting**:
  \[
  J_f = J + \epsilon_{1} ||W||_{1}
  \]
- **Hidden-unit clarification**:
  \[
  J_h = J + \epsilon_{2} \sum_j \min\{1-h_j, h_j\}
  \]
- **Selective forgetting (final pruning stage)**:
  \[
  J_s = J - \epsilon_{3} ||W'||_{1},\quad W'_{ij} = 
  \begin{cases} W_{ij}, & |W_{ij}|<\theta \\
  0, & \text{otherwise} \end{cases}
  \]

### 3.5. Layer Generation Criteria

For $k$ layers, compute:
\[
G_{WD} = \sum_{l=1}^k \alpha_{WD} WD^l
\]
\[
G_E = \sum_{l=1}^k \alpha_{E} E^l
\]
If $G_{WD} > \theta_{L1}$ and $G_E > \theta_{L2}$, the DBN grows by initializing a new RBM on top (parameters inherited).

[1807.03486]

## 4. Layer Generation and Hybrid Structural Learning

- **Layer Growth:** At the end of each epoch, global statistics (weighted sum of layerwise parameter variance and energy) are computed. If both exceed thresholds, a new layer is added and initialized by parameter inheritance from its parent.
- **RBM Width Adaptation:** Through neuron generation and pruning, each RBM layer dynamically fits the data complexity during pretraining.
- **Structural Learning with Forgetting:** SLF injects sparsity, restricts over-parameterization, and encourages hidden-unit interpretability—yielding layers that are both compact and extract explicit knowledge from data.
- **Integrated Process:** The entire architecture is thus shaped adaptively: width (neurons), depth (layers), and weight sparsity are co-optimized per dataset.

## 5. Computational Complexity and Stability

Each RBM's training step is $O(d \cdot h)$ per data vector, where $d$ is the visible dimension and $h$ the current number of hidden units. Adaptive structure adds $O(h)$ per batch for generation/annihilation checks. Layer generation overhead is negligible (per-epoch summations). Total cost up to a learned $K$-layer architecture is
\[
O\left( \sum_{l=1}^K \text{Epochs}_l \cdot N \cdot d_l \cdot h_l \right)
\]
Stability is dynamically monitored by the variance $WD^l$ and energy $E^l$ statistics in each layer; persistent high values in these quantities trigger structural growth, thus preventing underfitting and guiding self-organization [1807.03486].

## 6. Experimental Protocol and Results

**Datasets:** CIFAR-10 and CIFAR-100, with 50,000 training and 10,000 test 32×32 color images. ZCA whitening is applied to all inputs.

**Hyperparameters:** 
- Initial hidden units per layer: 300
- Mini-batch size: 100
- Learning rate: $\eta=0.1$
- Thresholds: $\theta_G \in \{0.05, 0.01\}$, $\theta_A$ chosen to prune underactive neurons
- Layer generation: $\theta_{L1}$, $\theta_{L2}$ set to yield 4–6 layers
- Forgetting coefficients: $\epsilon_1, \epsilon_2, \epsilon_3 \ll 1$

**Performance:**
- CIFAR-10: Up to 97.1% test accuracy ($\theta_G=0.01$, 5 layers), exceeding traditional DBN ($\sim 78.9\%$) and CNN baseline (96.5%).
- CIFAR-100: 81.3%, surpassing comparable CNN results (75.7%).
- The learned DBN for CIFAR-10 self-organized into layer sizes near [433, 1595, 369, 1462, 192]; model energy and error decrease monotonically as layers are added.

[1807.03486]

## 7. Significance and Applications

AdaBLDM provides a methodology for fully data-driven, architecture-agnostic training of deep generative models. By automating structure discovery at both the unit and layer level, it avoids the limitations of fixed-architecture models and reduces dependency on human hyperparameter selection. The hybridization of adaptive width (neuron-level structural learning), depth (layer generation), and global sparsification (structural forgetting) produces DBNs that are both compact and high performing. This framework is directly applicable to any domain that previously relied on hand-engineered DBN architectures, and experimental results demonstrate utility in image classification contexts. 

[1807.03486]

Source: https://www.emergentmind.com/topics/adabldm-algorithm