Hierarchical Nonnegative Matrix Factorization (HNMFk)

• Hierarchical Nonnegative Matrix Factorization (HNMFk) is a recursive method that decomposes nonnegative data into nested, interpretable latent factors.
• It employs binary or k-way splits and multilayer architectures using approaches like HALS to optimize stability and error reduction.
• HNMFk is applied in hyperspectral imaging, document clustering, face recognition, and population analysis, outperforming flat NMF methods.

Hierarchical Nonnegative Matrix Factorization (HNMFk) refers to a class of matrix factorization algorithms designed to extract multilevel or nested structure from high-dimensional, nonnegative data by recursively applying NMF in a hierarchical or multilayer fashion. In contrast to flat NMF, which provides a single nonnegative decomposition, HNMFk generates a tree or deep architecture, enabling the identification of increasingly abstract representations or clusters at each level. Applications span hyperspectral imaging, clustering, topic modeling, population survey analysis, and deep feature learning.

1. Mathematical Foundations of HNMFk

Hierarchical NMF builds on the classical NMF problem: given a nonnegative matrix $M\in\mathbb{R}_+^{m\times n}$ and rank $r<\min(m,n)$, find nonnegative factors $W\geq0$, $H\geq0$ such that

$\min_{W,H\geq0} \|M - W H\|_F^2.$

HNMFk extends this framework hierarchically, producing a sequence or tree of decompositions:

• Binary and $k$-way hierarchical splits: At each node (cluster or group), the data matrix or a submatrix is recursively factorized, often at a fixed low rank (typically 2 for binary, or $k$ for $k$-way splits) (Gillis et al., 2013).
• Multilayer deep hierarchies: In deep HNMFk, multiple layers are formed where each layer's encoding is then factorized at the next level, yielding

  $X \approx W^{(1)} W^{(2)} \cdots W^{(L)} H^{(L)}$

  for $L$ layers (Qiu et al., 2017, Yu et al., 2018, Will et al., 2023).

• Stability and stopping: Node splitting is often governed by explicit metrics, such as error reduction, feature similarity, or stability scores (Gillis et al., 2013, Ding et al., 2022).

This recursive structure differentiates HNMFk from ordinary NMF and enables the extraction of hierarchical latent structures.

2. Core Algorithms and Cluster-Splitting Strategies

Various HNMFk methods leverage different techniques for recursive splitting, hierarchical alternation, or multilayer construction:

• Rank-2 NMF Splitting: For binary hierarchical clustering, each cluster $r<\min(m,n)$0 is split by solving a rank-2 NMF subproblem:

  $r<\min(m,n)$1

  The subcluster assignment is based on the weight distribution in $r<\min(m,n)$2; a threshold is chosen to balance size and ensure stability to perturbations (Gillis et al., 2013).

• Selection of Split: At each iteration, the next cluster to split is chosen to maximize immediate total error reduction $r<\min(m,n)$3; precisely,

  $r<\min(m,n)$4

  where $r<\min(m,n)$5 denotes the largest singular value (Gillis et al., 2013).

• Block-Coordinate Descent / HALS: Hierarchical Alternating Least Squares (HALS or HNMFk) updates one factor column or block at a time, with each nonnegative least-squares (NNLS) subproblem solved in closed form. HALS alternates between iteratively updating columns of $r<\min(m,n)$6 and rows of $r<\min(m,n)$7 until convergence (Gillis et al., 2011).
• Accelerated HALS: Acceleration is accomplished by reusing expensive Gram matrix computations for multiple inner passes, implementing early stopping based on movement thresholds, which yields significant computational improvements while preserving convergence (Gillis et al., 2011).

3. Multilayer and Deep HNMFk Architectures

HNMFk generalizes from shallow binary splits to deep architectures wherein hierarchical representations are constructed over multiple layers:

• Deep nsNMF (dnsNMF): Stacks $r<\min(m,n)$8 layers of non-smooth NMF (nsNMF) to generate hierarchies:

  $r<\min(m,n)$9

  where $W\geq0$0 induces sparsity and localization, and global fine-tuning is performed after layer-wise pretraining (Yu et al., 2018).

• Deep Approximately Orthogonal NMF (DAONMF): Builds $W\geq0$1-layer factorizations with orthogonality penalties on each $W\geq0$2 to enforce clustering structure, yielding embedding hierarchies at multiple abstraction levels (Qiu et al., 2017).
• Neural NMF: Frames hierarchical multilayer NMF as a neural-net architecture, treating $W\geq0$3 as trainable weights and $W\geq0$4 as layer activations produced via NNLS, with backpropagation-derived updates and nonnegativity projection (Will et al., 2023).
• Population-based HNMF: Applies multi-start runs at each split and adopts stability criteria based on consensus (e.g., feature similarity among topic vectors), enabling the discovery of robust subgroup hierarchies in heterogeneous data (Ding et al., 2022).

Empirical results show that these deep architectures recover more abstract, discriminative codes with superior clustering and interpretability compared to shallow NMF.

4. Applications and Empirical Performance

HNMFk methods have been validated across multiple domains:

• Hyperspectral Image Clustering: HNMFk (H2NMF) achieves fast, accurate clustering and endmember extraction, demonstrating robustness to noise, illumination variation, and outliers (Gillis et al., 2013). In real data, H2NMF outperformed hierarchical $W\geq0$5-means, spherical $W\geq0$6-means, and flat NMF, attaining $W\geq0$7 accuracy for moderate noise and lower mean residual spectral angles.
• Face/Image Clustering: Deep HNMF variants (dnsNMF, DAONMF) consistently deliver higher clustering accuracy and normalized mutual information than flat NMF, nsNMF, graph-regularized NMF, and Deep Semi-NMF. For ORL, JAFFE, and Yale datasets, dnsNMF improved accuracy by 8–12% over shallow baselines (Yu et al., 2018, Qiu et al., 2017).
• Topic Modeling and Document Hierarchies: Neural NMF discovers layered topic structures in synthetic and real corpora, with superior reconstruction error and classification accuracy compared to HNMF and DeepNMF. Hierarchical decomposition enables interpretability at multiple levels, e.g., in the 20 Newsgroups dataset, neural NMF uncovers super-topics coherent with human-labeled divisions (Will et al., 2023).
• Population Structure Analysis: PHNMF accurately recovers ground-truth subgroup trees in synthetic survey data and produces interpretable population splits in real surveys, often exceeding 99% subgrouping accuracy (Ding et al., 2022).

5. Theoretical Guarantees and Complexity

Several theoretical properties underpin HNMFk:

• Exactness for Rank-2 and Separable Cases: Under pure-pixel or separability assumptions, the rank-2 NMF subproblem yields exact splits. If all columns sum to one and have rank two, SPA identifies the pure columns, and NNLS reconstructs their convex combinations exactly (Gillis et al., 2013).
• Nonnegativity Preservation: If the entrywise minimum of $W\geq0$8 exceeds its third singular value, the best rank-two approximation remains nonnegative, so the splitting process never leaves the positive orthant (Gillis et al., 2013).
• Computational Complexity: Each rank-two split and thresholding is $W\geq0$9; overall cost is $H\geq0$0 or $H\geq0$1 for balanced trees (Gillis et al., 2013). Deep architectures entail $H\geq0$2 per sweep, making HNMFk comparable in cost to shallow NMF for small $H\geq0$3 (Yu et al., 2018).
• Convergence: Block-coordinate approaches (HALS, ARkNLS) guarantee monotonic decrease of the global cost and convergence to stationary points, even with acceleration (Gillis et al., 2011, Chu et al., 2020).

6. Limitations, Extensions, and Open Problems

Notwithstanding advantages, HNMFk faces specific limitations:

• Split Criterion Sensitivity: The quality and balance of splits depend on criteria for threshold selection and stability. Empirical results suggest alternative or more adaptive criteria may improve performance on highly unbalanced or heterogeneous clusters (Gillis et al., 2013).
• No Theoretical Guarantees for $H\geq0$4 Splits: While rank-2 subproblems admit convex guarantees, extending exactitude to higher-rank splits or to general nonseparable data remains unresolved (Gillis et al., 2013).
• Scalability Concerns: While per-split cost is linear, the number of splits may grow rapidly in unbalanced data. Deep architectures with many layers can also introduce additional computational and tuning overhead.
• Incorporation of Regularization and Supervision: Methods such as Neural NMF incorporate penalties and allow semi-supervised extensions, while classic splitting approaches are usually unsupervised and lack such flexibility (Will et al., 2023).

Potential directions include integration of spatial regularization, adaptive splitting strategies, and further exploration of connections to deep autoencoders and neural networks for improved feature abstraction (Yu et al., 2018, Qiu et al., 2017).

7. Comparative Summary of HNMFk Methods

The following table contrasts principal HNMFk families as documented in the literature:

Method	Hierarchy Mechanism	Key Attributes / Best Use
H2NMF (HNMFk, (Gillis et al., 2013))	Recursive rank-2 NMF and cluster splitting	Fast, convex-geometric, balanced splits
HALS (HNMFk, (Gillis et al., 2011))	Block-coordinate descent in single/multilayer	Efficient NMF solver, supports acceleration
Deep nsNMF (Yu et al., 2018)	Stacked non-smooth NMF blocks	Deep, localized, sparse part-based features
DAONMF (Qiu et al., 2017)	Multi-layer with orthogonality penalty	Clean cluster-friendly hierarchical codes
Neural NMF (Will et al., 2023)	Multilayer NNLS with backprop gradient	Neural-net architecture, interpretable hierarchy
PHNMF (Ding et al., 2022)	Top-down splits using feature-stability	Robust, interpretable population structure

Each HNMFk variant is tailored to specific data modalities, theoretical priorities (e.g., separability, sparsity, interpretability), and computational constraints, enabling broad application in unsupervised learning, multiscale analysis, and interpretable representation discovery.