Hierarchical Multi-layer nsNMF

• The paper demonstrates that layered nsNMF improves reconstruction and clustering accuracy compared to shallow models.
• It employs a layer-wise pretraining and joint fine-tuning strategy using accelerated proximal-gradient optimization.
• The approach leverages smoothness constraints to enforce sparsity while building abstract, hierarchical feature representations.

Hierarchical Multi-layer Non-smooth Non-negative Matrix Factorization (nsNMF) extends the concept of Non-negative Matrix Factorization (NMF) by stacking multiple non-smooth NMF layers, thereby enabling the learning of hierarchical, parts-based representations from nonnegative data. In contrast to shallow NMF variants, this multi-layer architecture develops increasingly abstract feature hierarchies, combining localized, sparsity-controlled encoding at each layer. Empirical and theoretical investigations demonstrate that this framework is superior to shallow nsNMF for reconstruction, clustering, and classification tasks, particularly under feature dimension constraints (Yu et al., 2018, Song et al., 2013).

1. Mathematical Foundations

The base formulation of non-smooth NMF seeks a factorization $X \approx Z S H$, where $X\in\mathbb{R}^{p\times n}_+$, $Z\in\mathbb{R}^{p\times r}_+$, $H\in\mathbb{R}^{r\times n}_+$, and the smoothing matrix is $S = (1-\theta) I_r + \frac{\theta}{r} 1_r 1_r^T$ with $\theta\in[0,1]$. The minimization objective is

$\min_{Z,H\geq 0}\;\frac{1}{2}\|X - Z S H\|^2_F.$

In the hierarchical (multi-layer) extension, the $L$-layer architecture is expressed as

$$X \approx Z^{(1)} S^{(1)} Z^{(2)} S^{(2)} \cdots Z^{(L)} S^{(L)} H^{(L)},$$

or equivalently, as a layer-wise factorization chain: $$X \approx W_1 H_1,\quad H_1 \approx W_2 H_2,~\dots,~H_{L-1} \approx W_L H_L,$$ with $W_\ell = Z^{(1)} S^{(1)} \cdots Z^{(\ell)}$ for each layer $\ell$. Each layer thus learns a basis $W_\ell$ and code $H_\ell$ at increasing abstraction.

2. Hierarchical Feature Representation

The core motivation for hierarchical multi-layer nsNMF is the explicit discovery of feature hierarchies. In the tiered architecture:

• Layer 1 ($\ell=1$): Basis $W_1$ learns localized atomic features, e.g., pixels or edges for images, word co-occurrences for documents.
• Layer 2 ($\ell=2$): $W_2$ encodes more complex motifs, such as edge groupings or topic clusters.
• Higher Layers ($\ell\geq3$): Progressive abstraction yields composite features, e.g., facial organs from contours in images, broader topics from fine-grained document themes.

At each layer, the code $H^{(\ell)}$ is the nonnegative decomposition of $H^{(\ell-1)}$ over $W_\ell$, reinforcing the data’s nested and compositional structure. This explicit cascade enables re-use and recombination of lower-level representations, yielding richer, distributed encodings (Yu et al., 2018, Song et al., 2013).

3. Optimization Algorithms

The training process proceeds in two phases: layer-wise pretraining, followed by joint fine-tuning to minimize the end-to-end reconstruction error. For the deep nsNMF objective,

$$\min_{\{Z^{(\ell)}\},H^{(L)}\geq 0}~\frac{1}{2}\|X - Z^{(1)}S^{(1)}\cdots Z^{(L)}S^{(L)}H^{(L)}\|_F^2,$$

block-coordinate schemes are employed. Each block update leverages accelerated proximal-gradient (APG) steps with Nesterov momentum for convergence rate $O(1/k^2)$. The smoothing matrices $S^{(\ell)}$ at each layer modulate sparsity:

• Larger $\theta^{(\ell)}$: More smoothing on $H^{(\ell)}$, yielding sparser and more localized features.
• Initialization: Layer-wise, via NNDSVD or random+SVD, followed by layer stacking.
• Convergence: Achieved when relative objective change is below prescribed threshold (e.g., $10^{-4}$).

For the multi-layer algorithm in (Song et al., 2013), multiplicative update rules are derived for each $W^{(l)}$ and $H^{(l)}$, incorporating the smoothing regularization and backpropagated reconstruction errors through all layers. Interleaved “smoothing” ($H\leftarrow S H$) ensures persistent control over activation sparsity.

4. Theoretical Properties and Connections

Hierarchical multi-layer nsNMF yields distributed representations that, for any fixed code dimension $k$, achieve strictly improved upper bounds on reconstruction error over single-layer nsNMF. Under constraints on sparsity and component incoherence, hierarchical nsNMF codes exhibit provably higher Fisher discriminants.

A salient structural insight is the formal correspondence between deep nsNMF and a class of deep autoencoders: a dnsNMF model is equivalent to an “all-positive” autoencoder with tied, nonnegative weights and no bias terms, using nonnegativity and smoothness matrices to regularize activations. The forward recursion $H^{(\ell-1)} = Z^{(\ell)}S^{(\ell)}H^{(\ell)}$ and decoder unrolling align exactly with a ReLU autoencoder’s functional form but restricted to nonnegative parameters and outputs (Yu et al., 2018).

5. Hyperparameter Choices and Practical Guidance

Key hyperparameters and practical considerations include:

• Depth ($L$): 2–4 layers are effective for mid-scale image or document corpora; deeper architectures show diminishing returns.
• Layer Widths ($r_1 > r_2 > \cdots > r_L$): For facial images, typical settings are $r_1=100$–$200$, $r_2=50$–$100$, $r_L=K$, where $K$ is the number of clusters or code dimension.
• Smoothing Parameter ($\theta^{(\ell)}$): Values in $[0.3,0.9]$ are recommended; grid search per layer is standard to tune the sparsity/overlap trade-off.
• Normalization: Optional $\ell_2$-column normalization of each $Z^{(\ell)}$ after updates.
• Initialization and Fine-tuning: Pretrain layers individually with single-layer nsNMF, then perform joint optimization across all layers.

6. Empirical Results

Empirical studies across image and document domains consistently report substantial improvements for hierarchical multi-layer nsNMF over shallow baselines. On clustering tasks with face images (datasets: ORL, JAFFE, Yale), multi-layer nsNMF improves clustering accuracy (AC) and normalized mutual information (NMI) by 10–15 percentage points relative to single-layer NMF variants (Yu et al., 2018). In document classification (Reuters-21578) and digit recognition (MNIST), the multi-layer model achieves reduced reconstruction error and higher classification accuracy, with particularly pronounced gains as feature dimensionality decreases.

Method	NMF	nsNMF	GNMF	Deep Semi-NMF	Deep nsNMF
Avg AC (ORL)	72.9%	74.1%	66.3%	76.1%	84.9%
Avg NMI (ORL)	68.8%	70.4%	64.5%	71.4%	81.0%

The model uncovers multi-level semantic structure: for documents, subtopics coalesce at higher layers (“oil production,” “oil contracts,” and “oil refinery” to “oil”); for images, edge and contour features assemble into coherent parts (e.g., facial organs or digit prototypes) (Song et al., 2013). These hierarchical codes yield sparser reconstructions, improved class separability, and are empirically associated with higher Fisher discriminant ratios.

7. Significance and Applications

Hierarchical multi-layer nsNMF substantially augments the interpretability and abstraction capability of nonnegative matrix factorization. The method is particularly advantageous in scenarios with limited code dimensions, where the flexible combination and recombination of lower-layer features directly enable superior clustering, classification, and reconstruction. Its theoretical grounding and block-coordinate optimization guarantee convergence to stationary points, while carrying interpretability from NMF into deeper representation learning. The demonstrated correspondence with deep autoencoders positions multi-layer nsNMF as a bridge between interpretable matrix factorization and deep learning paradigms, with applicability in image analysis, topic modeling, and unsupervised clustering (Yu et al., 2018, Song et al., 2013).