---
title: 'Deep Semi-NMF: Hierarchical Matrix Factorization'
url: https://www.emergentmind.com/topics/deep-semi-nmf
type: topic
---

# Deep Semi-NMF: Hierarchical Matrix Factorization

Deep Semi-Non-negative Matrix Factorization (Deep Semi-NMF) is a hierarchical matrix factorization framework designed to recover interpretable, multi-level attribute representations from high-dimensional data. Extending the concept of Semi-NMF to a deep, multi-layer architecture, it models the generative structure of data as a product of stacked linear transformations culminating in non-negative latent factors. Unlike classical flat matrix factorization, Deep Semi-NMF captures hierarchies of attributes, with each layer producing a non-negative feature matrix interpreted as soft cluster memberships for latent factors. This approach allows the uncovering of complex, nested structure in datasets, particularly when clustering or class labels reflect multiple, overlapping factors of variation [1509.03248].

## 1. Model Architecture

Given a data matrix $X \in \mathbb{R}^{p \times n}$, Deep Semi-NMF factorizes $X$ as:
$$
X \approx Z_1 Z_2 \ldots Z_m H_m
$$
where $Z_i \in \mathbb{R}^{k_{i-1} \times k_i}$ (with $k_0=p$) are stacked “basis” matrices with mixed signs and $H_m \in \mathbb{R}_{\geq 0}^{k_m \times n}$ is a non-negative feature (“attribute”) matrix. Intermediate layers introduce feature matrices $H_1, \ldots, H_{m-1}$, each non-negative, yielding the layerwise structure:
\begin{align*}
X &\approx Z_1 H_1 \\
H_1 &\approx Z_2 H_2 \\
&\vdots \\
H_{m-1} &\approx Z_m H_m
\end{align*}
Each $H_i$ is interpreted as a soft cluster-membership matrix for $k_i$ latent attributes [1509.03248].

## 2. Objective Functions and Constraints

For the unsupervised (purely linear) form, the Deep Semi-NMF objective is:
$$
\min_{\{Z_1 \ldots Z_m, H_m \geq 0\}} C_{\text{deep}} = \frac{1}{2} \| X - Z_1 \ldots Z_m H_m \|_F^2
$$
Optionally, non-negativity constraints can be enforced on all intermediate $H_i$ matrices.

A trace-based reformulation is also valid:
$$
C_{\text{deep}} = \frac{1}{2} \operatorname{Tr}[ X^T X - 2X^T (Z_1 \ldots Z_m H_m) + (Z_1 \ldots Z_m H_m)^T (Z_1 \ldots Z_m H_m) ]
$$
When partial attribute labels are available, the semi-supervised extension, Deep WSF, augments the objective with graph Laplacian regularizers:
$$
C_{\text{WSF}} = C_{\text{deep}} + \frac{1}{2} \sum_{i=1}^m \lambda_i \operatorname{Tr}( H_i^T L_i H_i )
$$
where each $L_i$ is the Laplacian matrix built from available class labels for layer $i$, and $\lambda_i$ is a tuning hyperparameter [1509.03248].

## 3. Optimization Methods

Deep Semi-NMF utilizes a two-phase optimization scheme:

**A. Greedy Layer-wise Pre-training:**  
Each layer solves a two-factor Semi-NMF on the output of the previous layer:
- For $i = 1$ to $m$, factor $X_{i-1} \approx Z_i H_i$ (with $X_0 = X$).
- Alternate updates:
    - $Z_i \gets X_{i-1} H_i^{\dagger}$, where $\dagger$ denotes the Moore–Penrose inverse.
    - $H_i$ is updated multiplicatively to ensure $H_i \ge 0$:
      $$
      H_i \gets H_i \odot \sqrt{ \frac{ [Z_i^T X_{i-1}]^{+} + [Z_i^T Z_i]^{-} H_i }{ [Z_i^T X_{i-1}]^{-} + [Z_i^T Z_i]^{+} H_i } }
      $$
      with $A^+ = (|A| + A)/2$, $A^- = (|A| - A)/2$.

**B. Joint Fine-Tuning:**  
After pre-training, all factors are updated via alternating minimization:
- For each layer $i$, define $\Psi_i = Z_1 \ldots Z_{i-1}$ and $\tilde{H}_i$ (equal to $H_i$ if $i=m$, else $Z_{i+1} \ldots Z_m H_m$).
- $Z_i$-update (closed-form least squares):
  $$
  Z_i \gets \Psi_i^{\dagger} X \tilde{H}_i^{\dagger}
  $$
- $H_i$-update (multiplicative, preserves non-negativity):
  $$
  H_i \gets H_i \odot \sqrt{ \frac{ [\Psi_i^T X]^+ \tilde{H}_i^T + [\Psi_i^T \Psi_i]^- H_i }{ [\Psi_i^T X]^- \tilde{H}_i^T + [\Psi_i^T \Psi_i]^+ H_i } }
  $$
- Repeat until $C_{\text{deep}}$ convergence [1509.03248].

For Deep WSF, the $H_i$-update includes $\lambda_i$-weighted Laplacian regularization terms in numerator and denominator, leveraging partial supervision.

## 4. Semi-Supervised Extension: Deep WSF

When partial attribute-label supervision is present, Deep WSF (Deep Weakly Supervised Factorization) incorporates a graph-based smoothness term into each layer’s factorization. For samples with known memberships $y_j^{(i)}$ in classes at layer $i$, a similarity graph $W_i$ is constructed and its Laplacian $L_i = D_i - W_i$ (with $D_i$ diagonal) added to the loss as
$$
\text{Penalty}_i = \operatorname{Tr}( H_i^T L_i H_i )
$$
The resulting optimization uses the same multiplicative $H_i$ update rule as in Deep Semi-NMF, but adds $\lambda_i(W_i H_i)$ in the numerator and $\lambda_i(D_i H_i)$ in the denominator, promoting smoother, label-consistent attribute representations. The pretraining step for each layer is switched from standard Semi-NMF to WSF to integrate available label information from the outset [1509.03248].

## 5. Algorithmic Summary and Computational Complexity

The training process is as follows:

1. **Layer-wise Pre-training:**
    - Initialize $Z_i$, $H_i > 0$ (e.g., SVD-based).
    - For $i=1$ to $m$, run Semi-NMF (or WSF if supervised) on $H_{i-1} \approx Z_i H_i$.
    - Persist factors $Z_i$, $H_i$.
2. **Joint Fine-Tuning:**
    - Alternate $Z_i$ and $H_i$ updates for each layer until convergence.

Per-iteration computational complexity is $O(m[p n k + (p+n) k^2])$, with $k \approx \max k_i$ [1509.03248].

## 6. Empirical Results and Benchmarks

Deep Semi-NMF and Deep WSF have been validated on several standard face datasets:

| Dataset               | Samples   | Subjects | Attributes                    |
|-----------------------|-----------|----------|-------------------------------|
| XM2VTS                |   2,360   |   295    | 8 images/subject              |
| CMU PIE               |   2,856   |    68    | 42 illuminations/poses        |
| CMU Multi-PIE subset  |   7,905   |   147    | 5 poses, 6 expressions        |

Input features included raw pixels (all non-negative) and image-gradient–orientation (IGO) descriptors (mixed-sign). Baselines comprised NMF, Semi-NMF, GNMF, Multi-layer NMF, NeNMF, WSF, DNMF, and CNMF.

Performance metrics:
- Clustering: accuracy (AC), normalized mutual information (NMI), AUC of precision-recall.
- Downstream classification: linear SVM accuracy using learned $H$.

Notable empirical findings:
- Two-layer Deep Semi-NMF outperformed all single-layer and multi-layer NMF baselines in clustering by up to 15% AC gain.
- IGO features yielded enhanced separation relative to Semi-NMF.
- Supervised pretraining (Deep WSF on XM2VTS initializing Deep on CMU PIE) improved clustering accuracy by +5–8%.
- On CMU Multi-PIE, Deep WSF’s per-layer attributes most accurately classified the corresponding ground-truth factors: pose, expression, identity, each at different layers [1509.03248].

## 7. Hierarchical Attribute Representation and Interpretability

Each non-negative matrix $H_i$ in the deep hierarchy can be interpreted as a soft clustering over $k_i$ latent factors, corresponding to different attributes in the data. In multi-attribute face datasets, empirical assessment shows:
- Layer 1 (largest $k_1$): broad separation (e.g., head-pose clusters).
- Layer 2 (medium $k_2$): refinement into expression groups.
- Layer 3 (small $k_3$): subject identity clusters.

Columns of each $Z_i$ represent “basis portraits” or latent prototypes, with rows of $H_i$ indicating degrees of membership. Visualizing $H_i$ across layers reveals a staged “peeling away” of data variability: initial layers partition by high-variance attributes (e.g., pose), later layers resolve lower-variance ones (e.g., identity). This layered decomposition underwrites the method’s capacity to learn disentangled and attribute-aware representations [1509.03248].

Source: https://www.emergentmind.com/topics/deep-semi-nmf