A Sherman--Morrison--Woodbury approach to solving least squares problems with low-rank updates (2406.15120v2)

Abstract: We present a simple formula to update the pseudoinverse of a full-rank rectangular matrix that undergoes a low-rank modification, and demonstrate its utility for solving least squares problems. The resulting algorithm can be dramatically faster than solving the modified least squares problem from scratch, just like the speedup enabled by Sherman--Morrison--Woodbury for solving linear systems with low-rank modifications.

• The paper introduces WoodburyLS, which extends the SMW formula to efficiently update the Moore-Penrose pseudoinverse for least squares problems.
• The algorithm leverages precomputed factorizations to solve smaller over- and underdetermined systems, significantly reducing computational cost.
• Empirical results demonstrate up to a 130-fold speedup, confirming substantial efficiency gains for large-scale and frequently updated data.

A Sherman--Morrison--Woodbury Approach to Solving Least Squares Problems with Low-Rank Updates

The Sherman-Morrison-Woodbury formula is a well-established result in numerical computation, allowing for efficient updates to the inverse of a matrix subject to low-rank modifications. Despite its utility, until now, no extended application of this formula to the Moore-Penrose pseudoinverse has been practically employed for solving least squares problems. The paper under consideration addresses this gap by introducing a novel algorithm termed WoodburyLS, which adapts the Woodbury formula for updates to the pseudoinverse of full-rank rectangular matrices under low-rank modifications. The results demonstrate a significant computational efficiency in addressing modified least squares solutions.

Overview

The authors present a derivation generalizing the Sherman-Morrison-Woodbury formula for the Moore-Penrose pseudoinverse of full-rank rectangular matrices when perturbed by low-rank updates. The core contribution is a theoretical formula and an algorithm (WoodburyLS) which makes use of this development to solve least squares problems more efficiently than traditional methods, particularly in the context of large-scale data or frequent matrix modifications. This paper bridges gaps left by prior work such as Meyer's extension for rank-one updates and lays down a comprehensive approach for higher ranks.

Theoretical Contributions

The paper extends the foundation laid by the Sherman-Morrison-Woodbury formula, which addresses the inverse of square matrices. Specifically, it posits:

$$(A+UV^T)^\dagger = A^\dagger - MA^\dagger + (I-M)(A^TA)^{-1}VU^T,$$

where $M$ is derived from the components of $A$, $U$, and $V$. Crucially, this generalization provides a rank-$2r$ update to the pseudoinverse, anchoring the theoretical basis for computational savings.

Practical Implications and Algorithm

The algorithm WoodburyLS operates under the assumption that an initial factorization of $A$ (e.g., QR factorization) is already available. This precomputation allows leveraging the pseudoinverse update formula efficiently. It involves solving $2r$ overdetermined least squares problems and $2r$ underdetermined least squares problems, reducing computational complexity significantly over the naive approach of recomputing from scratch:

• Overdetermined Problems: $\min_x \|Ax - b\|_2$
• Underdetermined Problems: $\minimize \|x\|_2\ \text{subject to}\ A^Tx = b$

With a typical QR factorization, the computational cost is reduced by $O(n/r)$. The algorithm uses minimum-norm solutions for underdetermined systems, multiplying by the inverse of a $2r \times 2r$ matrix, and applying precomputed factors, resulting in a substantial efficiency gain.

Numerical Results

Empirical validation showcases the efficiency of WoodburyLS. The authors present results indicating up to 130-fold speedup over traditional methods for large matrices (e.g., $m =10^5, n = 1000$), reinforcing the practical utility of their theoretical advancements. This speedup is particularly notable for applications involving large data sets or scenarios where matrices undergo frequent low-rank modifications.

Future Developments

The methodological advancements in this paper pave the way for several future developments in computational mathematics and data science. Potential research avenues include:

• Extension to Rank-Deficient Cases: While the current work focuses on full-rank matrices, extending to rank-deficient scenarios could further broaden applicability.
• Iterative Methods Integration: Incorporating iterative solvers can enhance efficiency, especially for sparse matrices where direct factorization might be computationally prohibitive.
• Parallel Computations: As demonstrated, precomputation steps play a critical role; thus, parallel and distributed computing methods can further reduce runtimes and handle even larger data sizes.

Conclusion

The introduction of WoodburyLS represents a significant step forward in the efficient computation of least squares problems under low-rank updates. By extending the Sherman-Morrison-Woodbury formula to the pseudoinverse of rectangular matrices, the authors provide both theoretical and practical advancements, offering substantial computational savings. This development holds promise for a diverse array of applications in scientific computing and data science, where both the scale of data and the need for efficient recalculations are ever-increasing. This paper, through its rigorous derivation and validated performance, contributes a valuable tool for numerical computations and sets the stage for continued innovation in this area.