Deep Semi-NMF: Hierarchical Matrix Factorization

• Deep Semi-NMF is a hierarchical multi-layer matrix factorization framework that decomposes high-dimensional data into interpretable, non-negative soft cluster memberships.
• It extends traditional Semi-NMF by stacking linear transformations to uncover nested feature hierarchies and reveal complex, overlapping attributes.
• The method employs greedy layer-wise pretraining followed by joint fine-tuning with multiplicative update rules to improve clustering accuracy and classification performance.

Deep Semi-Non-negative Matrix Factorization (Deep Semi-NMF) is a hierarchical matrix factorization framework designed to recover interpretable, multi-level attribute representations from high-dimensional data. Extending the concept of Semi-NMF to a deep, multi-layer architecture, it models the generative structure of data as a product of stacked linear transformations culminating in non-negative latent factors. Unlike classical flat matrix factorization, Deep Semi-NMF captures hierarchies of attributes, with each layer producing a non-negative feature matrix interpreted as soft cluster memberships for latent factors. This approach allows the uncovering of complex, nested structure in datasets, particularly when clustering or class labels reflect multiple, overlapping factors of variation (Trigeorgis et al., 2015).

1. Model Architecture

Given a data matrix $X \in \mathbb{R}^{p \times n}$, Deep Semi-NMF factorizes $X$ as:

$X \approx Z_1 Z_2 \ldots Z_m H_m$

where $Z_i \in \mathbb{R}^{k_{i-1} \times k_i}$ (with $k_0=p$) are stacked “basis” matrices with mixed signs and $H_m \in \mathbb{R}_{\geq 0}^{k_m \times n}$ is a non-negative feature (“attribute”) matrix. Intermediate layers introduce feature matrices $H_1, \ldots, H_{m-1}$, each non-negative, yielding the layerwise structure: \begin{align*} X &\approx Z_1 H_1 \ H_1 &\approx Z_2 H_2 \ &\vdots \ H_{m-1} &\approx Z_m H_m \end{align*} Each $H_i$ is interpreted as a soft cluster-membership matrix for $k_i$ latent attributes (Trigeorgis et al., 2015).

2. Objective Functions and Constraints

For the unsupervised (purely linear) form, the Deep Semi-NMF objective is:

$$\min_{\{Z_1 \ldots Z_m, H_m \geq 0\}} C_{\text{deep}} = \frac{1}{2} \| X - Z_1 \ldots Z_m H_m \|_F^2$$

Optionally, non-negativity constraints can be enforced on all intermediate $X$0 matrices.

A trace-based reformulation is also valid:

$X$1

When partial attribute labels are available, the semi-supervised extension, Deep WSF, augments the objective with graph Laplacian regularizers:

$X$2

where each $X$3 is the Laplacian matrix built from available class labels for layer $X$4, and $X$5 is a tuning hyperparameter (Trigeorgis et al., 2015).

3. Optimization Methods

Deep Semi-NMF utilizes a two-phase optimization scheme:

A. Greedy Layer-wise Pre-training:

Each layer solves a two-factor Semi-NMF on the output of the previous layer:

• For $X$6 to $X$7, factor $X$8 (with $X$9).
• Alternate updates:
  • $X \approx Z_1 Z_2 \ldots Z_m H_m$0, where $X \approx Z_1 Z_2 \ldots Z_m H_m$1 denotes the Moore–Penrose inverse.
  • $X \approx Z_1 Z_2 \ldots Z_m H_m$2 is updated multiplicatively to ensure $X \approx Z_1 Z_2 \ldots Z_m H_m$3:

  $X \approx Z_1 Z_2 \ldots Z_m H_m$4

  with $X \approx Z_1 Z_2 \ldots Z_m H_m$5, $X \approx Z_1 Z_2 \ldots Z_m H_m$6.

B. Joint Fine-Tuning:

After pre-training, all factors are updated via alternating minimization:

• For each layer $X \approx Z_1 Z_2 \ldots Z_m H_m$7, define $X \approx Z_1 Z_2 \ldots Z_m H_m$8 and $X \approx Z_1 Z_2 \ldots Z_m H_m$9 (equal to $Z_i \in \mathbb{R}^{k_{i-1} \times k_i}$0 if $Z_i \in \mathbb{R}^{k_{i-1} \times k_i}$1, else $Z_i \in \mathbb{R}^{k_{i-1} \times k_i}$2).

• $Z_i \in \mathbb{R}^{k_{i-1} \times k_i}$3-update (closed-form least squares):

$Z_i \in \mathbb{R}^{k_{i-1} \times k_i}$4

• $Z_i \in \mathbb{R}^{k_{i-1} \times k_i}$5-update (multiplicative, preserves non-negativity):

$Z_i \in \mathbb{R}^{k_{i-1} \times k_i}$6

• Repeat until $Z_i \in \mathbb{R}^{k_{i-1} \times k_i}$7 convergence (Trigeorgis et al., 2015).

For Deep WSF, the $Z_i \in \mathbb{R}^{k_{i-1} \times k_i}$8-update includes $Z_i \in \mathbb{R}^{k_{i-1} \times k_i}$9-weighted Laplacian regularization terms in numerator and denominator, leveraging partial supervision.

4. Semi-Supervised Extension: Deep WSF

When partial attribute-label supervision is present, Deep WSF (Deep Weakly Supervised Factorization) incorporates a graph-based smoothness term into each layer’s factorization. For samples with known memberships $k_0=p$0 in classes at layer $k_0=p$1, a similarity graph $k_0=p$2 is constructed and its Laplacian $k_0=p$3 (with $k_0=p$4 diagonal) added to the loss as

$k_0=p$5

The resulting optimization uses the same multiplicative $k_0=p$6 update rule as in Deep Semi-NMF, but adds $k_0=p$7 in the numerator and $k_0=p$8 in the denominator, promoting smoother, label-consistent attribute representations. The pretraining step for each layer is switched from standard Semi-NMF to WSF to integrate available label information from the outset (Trigeorgis et al., 2015).

5. Algorithmic Summary and Computational Complexity

The training process is as follows:

1. Layer-wise Pre-training:

   • Initialize $k_0=p$9, $H_m \in \mathbb{R}_{\geq 0}^{k_m \times n}$0 (e.g., SVD-based).
   • For $H_m \in \mathbb{R}_{\geq 0}^{k_m \times n}$1 to $H_m \in \mathbb{R}_{\geq 0}^{k_m \times n}$2, run Semi-NMF (or WSF if supervised) on $H_m \in \mathbb{R}_{\geq 0}^{k_m \times n}$3.
   • Persist factors $H_m \in \mathbb{R}_{\geq 0}^{k_m \times n}$4, $H_m \in \mathbb{R}_{\geq 0}^{k_m \times n}$5.
2. Joint Fine-Tuning:
   • Alternate $H_m \in \mathbb{R}_{\geq 0}^{k_m \times n}$6 and $H_m \in \mathbb{R}_{\geq 0}^{k_m \times n}$7 updates for each layer until convergence.

Per-iteration computational complexity is $H_m \in \mathbb{R}_{\geq 0}^{k_m \times n}$8, with $H_m \in \mathbb{R}_{\geq 0}^{k_m \times n}$9 (Trigeorgis et al., 2015).

6. Empirical Results and Benchmarks

Deep Semi-NMF and Deep WSF have been validated on several standard face datasets:

Dataset	Samples	Subjects	Attributes
XM2VTS	2,360	295	8 images/subject
CMU PIE	2,856	68	42 illuminations/poses
CMU Multi-PIE subset	7,905	147	5 poses, 6 expressions

Input features included raw pixels (all non-negative) and image-gradient–orientation (IGO) descriptors (mixed-sign). Baselines comprised NMF, Semi-NMF, GNMF, Multi-layer NMF, NeNMF, WSF, DNMF, and CNMF.

Performance metrics:

• Clustering: accuracy (AC), normalized mutual information (NMI), AUC of precision-recall.
• Downstream classification: linear SVM accuracy using learned $H_1, \ldots, H_{m-1}$0.

Notable empirical findings:

• Two-layer Deep Semi-NMF outperformed all single-layer and multi-layer NMF baselines in clustering by up to 15% AC gain.
• IGO features yielded enhanced separation relative to Semi-NMF.
• Supervised pretraining (Deep WSF on XM2VTS initializing Deep on CMU PIE) improved clustering accuracy by +5–8%.
• On CMU Multi-PIE, Deep WSF’s per-layer attributes most accurately classified the corresponding ground-truth factors: pose, expression, identity, each at different layers (Trigeorgis et al., 2015).

7. Hierarchical Attribute Representation and Interpretability

Each non-negative matrix $H_1, \ldots, H_{m-1}$1 in the deep hierarchy can be interpreted as a soft clustering over $H_1, \ldots, H_{m-1}$2 latent factors, corresponding to different attributes in the data. In multi-attribute face datasets, empirical assessment shows:

• Layer 1 (largest $H_1, \ldots, H_{m-1}$3): broad separation (e.g., head-pose clusters).
• Layer 2 (medium $H_1, \ldots, H_{m-1}$4): refinement into expression groups.
• Layer 3 (small $H_1, \ldots, H_{m-1}$5): subject identity clusters.

Columns of each $H_1, \ldots, H_{m-1}$6 represent “basis portraits” or latent prototypes, with rows of $H_1, \ldots, H_{m-1}$7 indicating degrees of membership. Visualizing $H_1, \ldots, H_{m-1}$8 across layers reveals a staged “peeling away” of data variability: initial layers partition by high-variance attributes (e.g., pose), later layers resolve lower-variance ones (e.g., identity). This layered decomposition underwrites the method’s capacity to learn disentangled and attribute-aware representations (Trigeorgis et al., 2015).