Algorithms for nonnegative matrix factorization with the beta-divergence (1010.1763v3)

Abstract: This paper describes algorithms for nonnegative matrix factorization (NMF) with the beta-divergence (beta-NMF). The beta-divergence is a family of cost functions parametrized by a single shape parameter beta that takes the Euclidean distance, the Kullback-Leibler divergence and the Itakura-Saito divergence as special cases (beta = 2,1,0, respectively). The proposed algorithms are based on a surrogate auxiliary function (a local majorization of the criterion function). We first describe a majorization-minimization (MM) algorithm that leads to multiplicative updates, which differ from standard heuristic multiplicative updates by a beta-dependent power exponent. The monotonicity of the heuristic algorithm can however be proven for beta in (0,1) using the proposed auxiliary function. Then we introduce the concept of majorization-equalization (ME) algorithm which produces updates that move along constant level sets of the auxiliary function and lead to larger steps than MM. Simulations on synthetic and real data illustrate the faster convergence of the ME approach. The paper also describes how the proposed algorithms can be adapted to two common variants of NMF : penalized NMF (i.e., when a penalty function of the factors is added to the criterion function) and convex-NMF (when the dictionary is assumed to belong to a known subspace).

• The paper presents majorization-minimization (MM) and majorization-equalization (ME) algorithms that generalize and improve heuristic updates for NMF with the β-divergence.
• The paper rigorously constructs an auxiliary function to ensure monotonic descent and extends the methods to penalized and convex NMF variants.
• The paper demonstrates that the ME algorithm converges faster than standard methods, validated across synthetic data, audio spectrograms, and facial images.

Overview of Algorithms for Nonnegative Matrix Factorization with the β-divergence

This paper, authored by Cédric Févotte and Jérôme Idier, provides a detailed exploration and development of algorithms for nonnegative matrix factorization (NMF) leveraging the β-divergence as the cost function. The β-divergence, parameterized by a shape parameter β, generalizes multiple well-known divergences, including Euclidean distance (β=2), Kullback-Leibler (KL) divergence (β=1), and Itakura-Saito (IS) divergence (β=0).

Main Contributions

1. Majorization-Minimization (MM) Algorithm:
   • This algorithm produces multiplicative updates for NMF with the β-divergence, generalizing heuristic updates used in prior work.
   • For β in the interval [1,2], the MM algorithm coincides with standard heuristic updates. Outside this interval, the updates differ by a β-dependent power exponent.
   • The paper shows, via the construction of an auxiliary function, that these multiplicative updates can be seen as an outcome of a majorization-minimization scheme.
2. Majorization-Equalization (ME) Algorithm:
   • Introduces an alternative algorithm design that traverses constant level sets of the auxiliary function, akin to overrelaxation schemes, resulting in larger and more effective steps compared to standard MM algorithms.
   • This method demonstrates experimentally faster convergence rates.
3. Theoretical Analysis and Extensions:
   • Provides proofs for the monotonicity of heuristic algorithms for β in the range (0,1).
   • Extends MM and ME algorithms to variants of NMF, including penalized NMF and convex-NMF, establishing their broader applicability.

Key Numerical Results and Insights

• Convergence and Stability: The paper demonstrates that the ME algorithm converges faster than both the MM and heuristic algorithms across different values of β, confirming its effectiveness. Convergence is validated by tracking both cost values and Karush-Kuhn-Tucker (KKT) residuals.
• Practical Applications: The algorithms are tested on synthetic data, audio spectrograms for music transcription, and face image datasets. Results show significant improvements in convergence speed without compromising the accuracy or quality of the factorization.
• Implementation Efficiency: Despite the increased complexity of the ME updates, the computational overhead is marginal compared to the MM algorithm, making the proposed methods practical for large-scale problems.

Implications and Future Work

Practical Implications: The development and validation of these algorithms open avenues for their application in diverse fields including signal processing, image reconstruction, and data clustering. The flexibility offered by tuning the β parameter allows for tailored factors that improve specific performance metrics relevant to the application domain.

Theoretical Implications: The construction of a unified auxiliary function for different divergences and its role in ensuring monotonic descent algorithms provide a robust theoretical foundation for further research. This paper addresses several challenges in the convergence and stability of NMF algorithms, contributing significantly to the field.

Future Research Directions:

• Monotonicity Proofs for Heuristic Algorithms: For β outside the range (0,1), it is still an open question to theoretically guarantee the monotonicity of heuristic updates.
• Convergence Analysis: Establishing general convergence guarantees for NMF algorithms with the β-divergence, particularly in terms of obtaining solutions that meet KKT conditions, remains a fundamental challenge.
• Exploring Non-Multiplicative Updates: There is potential in further investigating non-multiplicative update methods and their performance in NMF, especially in handling penalty terms and constraints more efficiently.

In summary, the paper makes significant strides in the development and theoretical underpinning of NMF algorithms using the β-divergence, offering practical algorithms with verified performance enhancements and laying the groundwork for future theoretical and practical advancements in the field.