A Nonmonotone Gradient-Based Algorithm for Symmetric Nonnegative Matrix Factorization and Graph Clustering

Abstract: Symmetric nonnegative matrix factorization (Symmetric NMF) approximates a matrix as $WW^T$ with nonnegative rectangular factor $W$. It has broad applications in graph clustering and machine learning. In contrast to the NMF, projected gradient methods for the symmetric problem had been associated with slow convergence. To address this, we introduce SNMPBB, the first adaptation of nonmonotone projected Barzilai-Borwein methods to Symmetric NMF, demonstrating that gradient algorithms are significantly more effective than previously understood. We further extend SNMPBB to graph clustering using the graph Laplacian regularization (Graph-SNMPBB) and to large problems with low-rank approximations (LAI-SNMPBB). For all variants we prove global convergence to first-order stationary points and also that Barzilai-Borwein curvature information is preserved with randomized approximations. On synthetic data, SNMPBB achieves 6 times speedup over the alternative SymANLS for similar residuals, with advantages growing at higher ranks. Across six real-world clustering benchmarks, Graph-SNMPBB matches or exceeds SymANLS accuracy. Lastly, LAI-SNMPBB outperforms state-of-the-art LAI-SymPGNCG on 34 SuiteSparse matrices in both runtime and residual quality.

• The paper demonstrates that a nonmonotone projected Barzilai-Borwein gradient algorithm efficiently solves SymNMF, outperforming classical methods in speed and accuracy.
• It leverages penalty-based coupling and a nonmonotone line search to guarantee global convergence and robust handling of nonconvex objectives.
• Extensions to graph clustering with Laplacian regularization and low-rank approximations yield significant computational gains on large-scale datasets.

Nonmonotone Gradient-Based Algorithms for Symmetric Nonnegative Matrix Factorization and Graph Clustering

Introduction

Symmetric Nonnegative Matrix Factorization (SymNMF), which approximates a symmetric nonnegative matrix $V$ as $WW^T$ with $W\geq0$, is central to various applications in unsupervised machine learning, especially clustering for graphs and high-dimensional data. The classical Alternating Nonnegative Least Squares (SymANLS) method, while robust, introduces substantial computational overhead due to repeated least-squares solves. Recent approaches leverage randomization and low-rank ideas to enhance scalability but do not address the fundamental algorithmic limitations of gradient-based methods historically deemed unsuitable for SymNMF due to slow convergence.

This paper proposes SNMPBB: a nonmonotone projected Barzilai-Borwein (BB) gradient algorithm specifically adapted for SymNMF. It provides a rigorous convergence analysis, clarifies its extension to graph clustering (Graph-SNMPBB), and establishes synergy with randomized low-rank approximations (LAI-SNMPBB). Theoretical advances are matched with thorough empirical comparisons against state-of-the-art SymNMF solvers, demonstrating significant improvements in efficiency and accuracy, particularly in high-rank and large-scale settings.

SymNMF: Complexity and Algorithmic Landscape

SymNMF seeks to solve

$\min_{W \geq 0} \frac{1}{2}\|V - WW^T\|_F^2,$

where $V\in\mathbb{R}^{n\times n}_+$ is symmetric. Its nonconvex quartic nature and symmetry constraint yield challenges absent from standard NMF, including parameter identifiability and optimization landscape complexity.

Historical approaches include:

• Multiplicative updates (Lee & Seung): Stable for NMF but slow for SymNMF due to exacerbated curvature and intrinsic nonconvexity.
• Projected gradient descent (PGD): Traditionally abandoned for SymNMF due to poor convergence rates [Zhu et al., (Zhu et al., 2018)].
• SymANLS (Kuang et al.): Alternates between NNLS subproblems with symmetry enforced by a penalty term, robust but costly in dense settings.
• Low-rank Approximate Input (LAI) methods: Utilize sketching for efficiency at the expense of increased model bias [Hayashi et al., 10.1137/24M1638355].

Methodology

SNMPBB: Nonmonotone Projected BB for SymNMF

SNMPBB transposes the nonmonotone projected Barzilai-Borwein update scheme, well-studied in general NMF [Li & Shi, sym16020154], to the symmetric setting by considering a penalty-based coupling: $$f_\lambda(W,H) = \frac{1}{2}\|V - WH\|_F^2 + \frac{\lambda}{2}\|W - H^T\|_F^2.$$ Instead of forcibly equating $H = W^T$ at all times, symmetry is softly enforced via a quadratic penalty, preserving degrees of freedom essential for efficient gradient updates.

Each step comprises:

• A projected gradient subproblem with adaptive scaling by local curvature (via the BB two-point step-size approximation),
• A second-order-inspired search direction (from a further projected BB gradient step),
• A nonmonotone line search giving globalization and controlled non-monotonicity, inspired by Birgin et al.'s paradigms,
• Relaxation parameter $\varphi > 1$ amplifying the search direction.

This approach is theoretically justified by showing both convexity (in each subproblem) and Lipschitz continuity hold, ensuring global convergence to stationary points despite the underlying nonconvexity.

Graph-SNMPBB: Regularization for Graph Clustering

Real-world graph clustering often requires more than vanilla SymNMF to capture cluster structures, especially with noise and sparse connections. Graph-SNMPBB extends SNMPBB via a graph Laplacian regularization: $$f_{\lambda, \gamma}(W,H) = \frac{1}{2}\|V - WH\|_F^2 + \frac{\lambda}{2}\|W - H^T\|_F^2 + \frac{\gamma}{2}\mathrm{tr}(HLH^T),$$ where $L$ is the normalized Laplacian. The Laplacian smoothness ensures that learned factors correspond to connected subgraphs, enhancing clustering quality.

LAI-SNMPBB: Low-Rank Input Acceleration

For high-dimensional $WW^T$0, SNMPBB operates on a low-rank sketch $WW^T$1 computed by randomized SVD or QR-based methods. All subsequent multiplies and gradient evaluations use $WW^T$2, $WW^T$3 (with $WW^T$4 of rank much less than $WW^T$5), changing the data term but provably not the BB step-size or the key curvature information. The gradient bias resulting from the approximation is controlled by the sketch accuracy $WW^T$6.

Theoretical Results

• Convergence: SNMPBB guarantees global convergence to first-order stationary points for the penalty-based objective, leveraging the convexity/Lipschitz framework of [Li & Shi, sym16020154]. The coupling via penalty ensures that solutions are approximately symmetric for sufficiently large $WW^T$7.
• Robustness to Regularization: Addition of graph regularization ($WW^T$8) does not affect the convexity or Lipschitz continuity of subproblems, preserving convergence properties.
• LAI Robustness: The BB step-size remains unaffected by the low-rank approximation error, since the curvature matrix with fixed $WW^T$9 is unchanged (the bias $W\geq0$0 cancels in difference steps).

Empirical Results

Synthetic Matrices

Across a suite of dense synthetic problems, SNMPBB achieves matching or lower residuals compared to SymANLS and SymNewton with a 6x speedup for high-rank ($W\geq0$1) factorizations. This advantage increases with rank, underscoring the scalability of first-order approaches.

Graph Clustering Benchmarks

On real-world benchmarks (COIL20, Isolet1, MNIST, ORL, Reuters-21578, TDT2), Graph-SNMPBB:

• Matches or exceeds accuracy of SymANLS and PGD,
• Converges in significantly less wall-clock time,
• Excels particularly on high-dimensional or sparse matrices, leveraging its $W\geq0$2 cost (versus $W\geq0$3 for SymANLS).

Without Laplacian regularization, clustering accuracy deteriorates substantially, highlighting the necessity of the graph-structured penalty.

SuiteSparse and LAI Comparisons

On 34 SuiteSparse matrices, LAI-SNMPBB consistently dominates LAI-SymPGNCG in both runtime and residual quality (70% of cases at best ratio). Limiting the number of inner iterations (3-5) further boosts performance, acting as an implicit regularizer and avoiding overfitting to the low-rank surrogate—a phenomenon consistent with findings in iterative regularization theory.

Web of Science (WOS) text clustering yields ARI and residuals on par with LAI-SymPGNCG, but in much less computational time.

Practical and Theoretical Implications

The introduction of SNMPBB and its extensions refutes the view that projected gradient methods are ill-suited for SymNMF. When paired with adaptive curvature scaling, nonmonotone globalization, and appropriate regularization, they are both efficient and highly competitive in solution quality, especially for high-rank and large-scale settings common in graph and document clustering.

On the theoretical side, the results generalize recent progress in first-order non-monotonic methods and clarify that low-rank surrogate approaches preserve key algorithmic invariants, opening the door for further algorithmic innovation. Empirically, these methods offer immediate practical utility due to easily parallelizable routines, avoidance of expensive factorizations, and favorable scaling.

Future research directions include principled strategies for hyperparameter choices (penalty weights, regularization), adaptive selection of rank in LAI, and tighter integration with domain-specific preprocessing (e.g., graph kernel design).

Conclusion

SNMPBB and its graph-regularized and low-rank input variants provide a computationally viable, theoretically sound framework for SymNMF and graph clustering. Through a combination of penalty-based coupling, BB step-size adaptation, nonmonotone globalization, and regularization, these methods achieve superior runtime and solution quality in both synthetic and real datasets, challenging the established reliance on costly second-order and alternating least-squares methods. These findings have direct implications for scalable clustering, graph mining, and high-dimensional data analysis.