---
title: 'Deep WSF: Weakly-Supervised Deep Semi-NMF'
url: https://www.emergentmind.com/topics/weakly-supervised-deep-semi-nmf-deep-wsf
type: topic
---

# Deep WSF: Weakly-Supervised Deep Semi-NMF

Weakly-Supervised Deep Semi-NMF (Deep WSF) is a multi-layer matrix factorization framework designed to learn hierarchical and attribute-specific representations from partially labeled data. This paradigm builds on the classical Semi-Nonnegative Matrix Factorization (Semi-NMF) and incorporates partial prior information using weak supervision, extending factorization into multiple nonnegative layers. Deep WSF simultaneously enables unsupervised learning of hidden semantic features and explicit encoding of available label information, supporting both clustering and classification across complex, multi-attribute datasets [1509.03248].

## 1. Model Formulation and Layered Factorization

At its core, Deep WSF operates on a data matrix $X \in \mathbb{R}^{p \times n}$, where $p$ is the feature dimension and $n$ is the number of samples. The factorization proceeds through $m$ layers:

$$
X \approx Z_1 Z_2 \cdots Z_m H_m
$$

- $Z_i \in \mathbb{R}^{k_{i-1} \times k_i}$: linear transformation (basis) matrices at layer $i$, with $k_0 = p$ and $k_m$ the dimension of the final code.
- $H_i \in \mathbb{R}^{k_i \times n}$: activation (code) matrices; a nonnegativity constraint $H_i \geq 0$ is imposed at all layers.

Each intermediate layer reconstructs its input as $H_{i-1} \approx Z_i H_i$, giving a hierarchical structure analogous to deep networks. Optional entrywise nonlinearities $g(\cdot)$ can be inserted, making $H_{i-1} \approx g(Z_i H_i)$, but the principal model is linear in the cited work.

For weak supervision, Deep WSF is equipped to integrate partial label information on one or more attributes at each layer, modeled via graph Laplacian regularization [1509.03248].

## 2. Objective Function and Regularization

The full optimization criterion is

$$
L_{Deep\,WSF} = \frac{1}{2} \| X - Z_1 Z_2 \cdots Z_m H_m \|_F^2 + \frac{1}{2} \sum_{i=1}^m \lambda_i \operatorname{Tr}( H_i^T L_i H_i )
$$

Subject to: $H_i \geq 0$ for all $i$.

- **Reconstruction Loss**: $\| X - Z_1 Z_2 \cdots Z_m H_m \|_F^2$ enforces a compact encoding for $X$.
- **Graph Laplacian Regularization**: Each $L_i$ is a Laplacian derived from partial labels or attribute similarity graphs at layer $i$; it penalizes divergence of low-dimensional codes $H_i$ for samples known (or presumed) to share an attribute.

If a nonlinearity $g$ is used, the reconstruction loss generalizes accordingly. The regularization parameter $\lambda_i$ controls the influence of supervision at each layer and is typically selected in the range $[10^{-4}, 10^{-2}]$.

## 3. Weak Supervision via Attribute Graphs

At each layer, weak supervision is achieved by constructing an adjacency graph $W_i$ reflecting known partial labels for the relevant attribute. The Laplacian is then $L_i = D_i - W_i$, where $D_i$ is the corresponding degree matrix.

- $W_{i,j} = 1$ if samples $i$ and $j$ share a known label for the supervised attribute at layer $i$, and $0$ otherwise.
- The penalty $\operatorname{Tr}( H_i^T L_i H_i )$ becomes a sum over $W_{jk} \| h_{i, j} - h_{i, k} \|^2$, encouraging codes to cluster for must-linked items.

In datasets with multiple known attributes (e.g., identity, pose, expression), Deep WSF can be configured so that each layer encodes a representation specialized for one attribute, with separate graphs and Laplacians at each level.

## 4. Optimization Algorithm and Practicalities

Training proceeds in two stages:

- **Greedy Layerwise Pre-training**: For each layer $i$, optimize the single-layer WSF subproblem:
  $$
  \min_{Z_i, H_i \geq 0} \frac{1}{2}\| H_{i-1} - Z_i H_i \|_F^2 + \frac{1}{2}\lambda_i \operatorname{Tr}( H_i^T L_i H_i )
  $$
  using multiplicative updates for $H_i$ and least-squares or pseudo-inverse for $Z_i$.

- **Global Fine-tuning**: Alternately update all $\{Z_i, H_i\}$ by:
  - $Z_i \gets$ least-squares solution from reconstructed code.
  - $H_i$ via component-wise multiplicative update:
    $$
    H_i \leftarrow H_i \odot \sqrt{
      \frac{A^{pos} + B^{neg} H_i + \lambda_i (H_i L_i)^{neg}}
           {A^{neg} + B^{pos} H_i + \lambda_i (H_i L_i)^{pos}}
    }
    $$
    where $A = \Psi_i^T X$, $B = \Psi_i^T \Psi_i$, and $\Psi_i = Z_1 Z_2\cdots Z_{i-1}$.
  - Optionally, renormalize to keep $\operatorname{Tr}( H_i^T L_i H_i )$ bounded.

The stopping criterion is typically a small relative objective change or a fixed iteration count; settings of 500–1,000 iterations are reported. Initialization is typically via SVD-based heuristics (NNDSVD or Gillis–Glineur).

## 5. Empirical Evaluation and Attribute Decoupling

In experimental settings on face datasets (XM2VTS, CMU-PIE, CMU-Multi-PIE), Deep WSF demonstrates statistically significant improvements in clustering accuracy (AC) and classification against Semi-NMF and alternative nonnegative or semi-supervised matrix factorization methods.

- For example, on XM2VTS (final layer dimension 40), Semi-NMF achieves AC$\approx$0.61, while Deep Semi-NMF yields AC$\approx$0.68. Using Image Gradient Orientation features, Deep models reach AC$\approx$0.77 (vs. Semi-NMF 0.63) [1509.03248].
- In the three-attribute classification on CMU-Multi-PIE, Deep WSF learns $H_1, H_2, H_3$ optimized for pose, expression, and identity, outperforming all previous semi-supervised NMF variants on identity classification by $\sim$10%, with attribute-specific accuracies of 100%, 82.9%, and 65.2%, respectively.
- Supervised pre-training on one dataset can transfer beneficially to another, as shown by AC improvements from 0.56 to 0.62 on CMU-PIE after pre-training on XM2VTS.

## 6. Computational Complexity and Implementation Guidelines

Reported computational complexity for Deep WSF (in the linear model) is:
- **Pre-training**: $O(m t (p n k + n k^2 + p^2 k + k n^2))$
- **Fine-tuning**: $O(m t_f (p n k + (p + n) k^2))$
where $k = \max_i k_i$, and $t$, $t_f$ are the number of iterations for pre-training and fine-tuning, respectively.

Key guidelines:
- Number of layers $m$ typically set to 2 or 3.
- Hidden sizes $k_i$ are dataset- and attribute-dependent, e.g., $k_1=625$, $k_2=20$–70.
- Regularization parameters $\lambda_i$ tuned via validation, recommended in $[10^{-4}, 10^{-2}]$ for partial supervision.
- Careful initialization is critical for convergence and stability.

## 7. Significance and Distinct Features

Deep WSF enables learning of deep, layered representations that are explicitly aligned with weak, attribute-level supervision, providing a principled methodology for capturing both global and attribute-specific structure in complex datasets [1509.03248]. Its layerwise Laplacian regularization fosters disentanglement along known axes of variability, in contrast to flat NMF methods. Empirical results show superior clustering and classification, with robustness to mixed or partial labels. Deep WSF also supports multi-attribute learning, yielding layerwise representations specialized for each attribute, a capability not present in conventional shallow factorization frameworks. 

A plausible implication is that Deep WSF stands as a foundation for future research on deep factorizations with multi-attribute or graph-based weak supervision, and it continues to inform more modern deep semi-NMF frameworks employing more advanced prior or label constraints [2009.03714, 1509.03248].

Source: https://www.emergentmind.com/topics/weakly-supervised-deep-semi-nmf-deep-wsf