---
title: 'Deep Semi-NMF: Hierarchical Matrix Factorization'
url: https://www.emergentmind.com/topics/deep-semi-non-negative-matrix-factorization-deep-semi-nmf
type: topic
---

# Deep Semi-NMF: Hierarchical Matrix Factorization

Deep semi-nonnegative matrix factorization (Deep semi-NMF) generalizes convex and semi-convex matrix factorization to deep or multilayer settings, incorporating hierarchical representation learning under constraints of (partial) non-negativity. Deep semi-NMF enables the discovery of interpretable and discriminative latent structures across multiple abstraction levels in data, with applications in clustering, classification, collaborative filtering, and neural network weight estimation. Layer stacking, elementwise nonlinearity, and alternating optimization define typical methodological frameworks.

## 1. Formulation and Theoretical Foundations

Deep semi-NMF extends shallow semi-NMF, which factorizes a real matrix $X\in\mathbb{R}^{m\times n}$ as $X\approx ZH$ with $Z\in\mathbb{R}^{m\times k}$ unconstrained and $H\in\mathbb{R}_+^{k\times n}$ nonnegative. The deep (multilayer) formulation is:

\[
X \approx W^{(1)}W^{(2)} \cdots W^{(L)} H^{(L)}
\]
where $W^{(1)}\in\mathbb{R}^{m\times k_1}$, $W^{(2)}\in\mathbb{R}^{k_1\times k_2}$, ..., $W^{(L)}\in\mathbb{R}^{k_{L-1}\times k_L}$, and $H^{(L)}\in\mathbb{R}_+^{k_L\times n}$. Intermediate matrices $H^{(\ell)}$ (for $1\leq\ell\leq L$) are defined recursively:
\[
H^{(\ell-1)} = W^{(\ell)}H^{(\ell)}, \quad H^{(0)} := X, \quad H^{(\ell)}\geq 0
\]
This structure imposes nonnegativity on the final layer's representations, with unconstrained factors otherwise, promoting “soft” cluster assignments at each level [1509.03248].

Nonlinearity can be interleaved:
\[
C_{\text{deep}}^* = \tfrac12 \| X - W^{(1)}g(W^{(2)}g(\cdots g(W^{(L)}H^{(L)}))) \|_F^2
\]
where $g(\cdot)$, e.g. ReLU or softplus, is applied elementwise [1710.05613].

In supervised deep semi-NMF for deep neural networks, the model augments this decomposition:
\[
\min_{\substack{W_1,\dots,W_d;\\Z_1,\dots,Z_{d-1}\geq 0}} \frac12\|Y - W_dZ_{d-1}\|_F^2 + \sum_{i=1}^{d-1}\frac12\|Z_i - f(W_i Z_{i-1})\|_F^2
\]
with $Z_0 = X$, $f(\cdot)$ typically ReLU. The first term is a semi-NMF, subsequent summands are nonlinear semi-NMFs [1605.04639].

## 2. Optimization Algorithms and Layerwise Procedures

Alternating minimization (“block-coordinate descent”) is fundamental. Greedy layerwise pre-training initializes $(W^{(\ell)},H^{(\ell)})$ by sequential (semi-)NMF or NMF on representations $H^{(\ell-1)}$ [1509.03248]. Fine-tuning proceeds by alternating updates over all factor matrices:

For the linear case, $W^{(\ell)}$ is solvable in closed form:
\[
W^{(\ell)} \leftarrow [\Psi^{(\ell)\dagger} X \tilde{H}^{(\ell)\dagger}]
\]
where $\Psi^{(\ell)}=W^{(1)}\cdots W^{(\ell-1)}$, $\tilde{H}^{(\ell)}$ is the partial right product, and $\dagger$ denotes the Moore–Penrose pseudoinverse.

$H^{(\ell)}$ update (multiplicative for linear case):
\[
H^{(\ell)} \leftarrow H^{(\ell)} \odot
\sqrt{\frac{[A]_+ + [B]_ - H^{(\ell)}}{[A]_- + [B]_+ H^{(\ell)}}}
\]
with $A=\Psi^{(\ell)\top} X$ and $B=\Psi^{(\ell)\top}\Psi^{(\ell)}$ [1509.03248].

For nonlinear objectives or activation functions, projected/stationary gradient iterations are used, maintaining $H^{(\ell)}\geq 0$. In supervised deep semi-NMF, stationary iteration updates align model weights with the nonlinear layerwise prediction (e.g., $W_i^{(s+1)} \gets W_i^{(s)} + \omega R_s (Z_{i-1})^\dagger$) and projection ensures nonnegativity of $Z_{i-1}$ [1605.04639].

Autoencoder-style (pre-)training is often used for initialization: standard NMF on each layer's output with subsequent nonlinear encoder fit [1605.04639].

## 3. Model Variants and Nonlinearity

Nonlinear deep semi-NMF structures insert elementwise nonlinearities between factorizing linear layers. In collaborative filtering, deep semi-NMF can apply nonlinearity only on the item side:
\[
R \approx B + P_1 Q_1^+,\quad Q_1^+ \approx g(S_2 Q_2^+)
\]
Yet the “interaction” remains linear in the topmost non-negative features. Stacking more than two nonlinear layers has been empirically shown to increase test RMSE, indicating diminishing returns for depth [1710.05613].

In deep neural network weight estimation, the nonlinear semi-NMF paradigm is leveraged for end-to-end layer-wise weight learning without explicit gradient backpropagation, using alternating minimization for both semi-NMF and nonlinear semi-NMF objectives [1605.04639].

The semi-supervised extension incorporates attribute information via Laplacian regularizers, enabling label propagation and enforcing smoothness within each $H^{(\ell)}$ based on partially known attributes (“Deep WSF”) [1509.03248].

## 4. Empirical Performance and Applications

### Clustering and Classification

Deep semi-NMF demonstrates strong clustering accuracy (AC), normalized mutual information (NMI), and SVM-based classification accuracy, consistently outperforming one-layer NMF/Semi-NMF, graph-regularized NMFs, and classical multi-layer NMF on standard face datasets (CMU PIE, XM2VTS, Multi-PIE), with gains (e.g., ACM2 PIE: Deep Semi-NMF ~54% vs Semi-NMF ~50% AC) confirmed across both pixel and image-gradient features [1509.03248].

Deep WSF further improves clustering/classification via multi-attribute Laplacian regularization, achieving significant advances in attribute-specific accuracies (e.g., pose, expression, identity) [1509.03248].

### Deep Neural Networks

Deep semi-NMF exhibits supervised performance within 0.1–0.2% of backpropagation-trained neural networks for MNIST and CIFAR-10 classification, with convergence times per epoch comparable (within 5–10%) to standard BP frameworks and improved robustness to hyperparameter (e.g., step-size) settings [1605.04639].

### Collaborative Filtering

In recommender systems, nonlinear deep semi-NMF attains lower RMSE than both shallow NMF and more complex deep matrix factorization approaches. For FilmTrust, MovieLens 100K, and Amazon Music, NSNMF (ReLU + bias) achieved the lowest RMSE in each case, e.g., 0.788 (FilmTrust), 0.887 (MovieLens 100K), and 0.836 (Amazon Music). NSNMF's item clustering quality (WCSS) is on par with deep MF models [1710.05613].

## 5. Interpretability, Hierarchical Structure, and Attribute Discovery

Each $H^{(\ell)}$ in deep semi-NMF corresponds to a non-negative, interpretable latent representation—akin to soft-clustering—at abstraction level $\ell$. This hierarchical modeling enables unsupervised discovery of data attributes such as pose, expression, or identity in face datasets, with the hierarchy empirically matching “known” attributes when available [1509.03248]. Layer stacking combined with nonnegativity endows final features with increased discriminativeness and interpretability relative to classical deep autoencoders, where both sides of the factorization may be unconstrained [1710.05613].

In collaborative filtering, the restriction to item-side depth (while user features remain linear) preserves identifiability of item factors and reduces overfitting risk, with interpretability retained in top-layer nonnegative representations [1710.05613].

## 6. Methodological Comparisons and Limitations

Relative to deep MF models with unconstrained representations (e.g., deep autoencoders, multilayer perceptrons), deep semi-NMF restricts nonnegativity to specific matrix blocks, maintaining linearity in the outer product and allowing “part-based” or clustering interpretation at the final layer or after each intermediate step. Deep semi-NMF differs from deep NMF approaches by its ability to handle mixed-sign input data and retain soft cluster-weights interpretation.

The effectiveness of deep semi-NMF is contingent upon suitable selection of depth ($L$) and layer widths $(k_1,\dots,k_L)$, which remain heuristic. Training complexity increases with layer count, and stacking beyond two layers provides diminishing or negative returns in standard tasks (notably in collaborative filtering) [1710.05613]. Layerwise greedy pretraining is critical in navigating the highly nonconvex loss landscape; without it, final representations may underperform. Laplacian regularization for semi-supervised deep semi-NMF improves attribute-disentanglement but introduces additional hyperparameters [1509.03248].

## 7. Future Directions

Extensions under investigation include multilinear/tensor generalizations, application to heterogeneous modalities (e.g., speech), and the development of more advanced nonconvex optimization algorithms to manage the scalability and depth of deep semi-NMF architectures. Further analysis of nonnegativity's inductive bias at deep layers, as well as its utility in domains beyond vision and recommendation, presents an open research direction [1509.03248, 1605.04639, 1710.05613].

Source: https://www.emergentmind.com/topics/deep-semi-non-negative-matrix-factorization-deep-semi-nmf