Sparse Inverse Covariance Matrix Estimation Using Quadratic Approximation (1306.3212v1)

Abstract: The L1-regularized Gaussian maximum likelihood estimator (MLE) has been shown to have strong statistical guarantees in recovering a sparse inverse covariance matrix, or alternatively the underlying graph structure of a Gaussian Markov Random Field, from very limited samples. We propose a novel algorithm for solving the resulting optimization problem which is a regularized log-determinant program. In contrast to recent state-of-the-art methods that largely use first order gradient information, our algorithm is based on Newton's method and employs a quadratic approximation, but with some modifications that leverage the structure of the sparse Gaussian MLE problem. We show that our method is superlinearly convergent, and present experimental results using synthetic and real-world application data that demonstrate the considerable improvements in performance of our method when compared to other state-of-the-art methods.

• The paper presents QUIC, a novel Newton-based method for efficient sparse inverse covariance matrix estimation in high-dimensional settings.
• QUIC employs a second-order optimization algorithm with quadratic approximations and improved coordinate descent, significantly reducing computational complexity.
• Empirical results demonstrate QUIC's superlinear convergence and superior scalability on both synthetic and real-world datasets compared to existing algorithms.

Sparse Inverse Covariance Matrix Estimation Using Quadratic Approximation

This paper presents a novel method called QUIC for estimating sparse inverse covariance matrices in high-dimensional settings, based on the ℓ1-regularized Gaussian MLE framework, which is vital for inferring the graph structure of Gaussian Markov Random Fields (GMRFs). QUIC employs a Newton-based second-order optimization algorithm with quadratic approximations, demonstrating superlinear convergence compared to existing methods such as ALM, glasso, PSM, SINCO, and IPM.

In high-dimensional statistical problems, estimating the inverse covariance matrix (also known as the precision matrix) is central, particularly when the number of parameters far exceeds the number of observations. This problem is prominent in applications like gene network inference, fMRI brain connectivity, and social network analysis. Traditional methods tend to scale poorly under these conditions due to sub-linear convergence rates, which motivate the need for more efficient algorithms.

The main contribution of this work is the QUIC algorithm, which frames the sparse inverse covariance estimation as a Lasso problem and solves it using coordinate descent techniques. The key innovation lies in exploiting the symmetry and specific structure of the problem, reducing the computational complexity of coordinate descent updates from O(p²) to O(p) by effectively caching computations and focusing on a subset of free variables. This approach distinguishes QUIC from other second-order methods like the Projected Quasi-Newton (PQN) and inexact interior point methods, which tend to be computationally intensive for large-scale data.

Empirical results demonstrate QUIC's superior performance in both synthetic and real-world datasets, achieving faster convergence rates than first-order methods and being able to efficiently handle an impressive scale of matrices, where other solvers experience performance bottlenecks. These include chain and random sparsity patterns in inverse covariance matrices, depicting QUIC's robustness and scalability.

Furthermore, the research emphasizes the adaptive adjustment of coordinate descent step numbers in QUIC, which initially uses fewer steps to achieve quick progress and later increases for more precise convergence. This flexibility in trade-off management contributes to its computational expediency.

Theoretical contributions include detailed convergence analysis and proofs of the global and quadratic convergence of QUIC. The algorithm is also shown to adaptively identify block-diagonal structures in the covariance matrix, effectively reducing problem size and enhancing computational efficiency in block-diagonal cases.

This work directly impacts both theoretical developments and practical implementations of sparse estimation in high-dimensional statistics. It provides foundational elements for future explorations into enhanced computational techniques for statistical learning models, especially those involving complex dependency structures. The insights gained may aid in developing more generalized methods that can further exploit sparsity and structure in various domains of AI. Future work could explore extensions of these techniques beyond Gaussian models to other distributions and use models that handle missing data natively or accommodate larger varietal inputs, broadening the potential applications of QUIC-based methods.