Optimization without Retraction on the Random Generalized Stiefel Manifold (2405.01702v3)

Abstract: Optimization over the set of matrices $X$ that satisfy $X^\top B X = I_p$, referred to as the generalized Stiefel manifold, appears in many applications involving sampled covariance matrices such as the canonical correlation analysis (CCA), independent component analysis (ICA), and the generalized eigenvalue problem (GEVP). Solving these problems is typically done by iterative methods that require a fully formed $B$. We propose a cheap stochastic iterative method that solves the optimization problem while having access only to random estimates of $B$. Our method does not enforce the constraint in every iteration; instead, it produces iterations that converge to critical points on the generalized Stiefel manifold defined in expectation. The method has lower per-iteration cost, requires only matrix multiplications, and has the same convergence rates as its Riemannian optimization counterparts that require the full matrix $B$. Experiments demonstrate its effectiveness in various machine learning applications involving generalized orthogonality constraints, including CCA, ICA, and the GEVP.

• The paper introduces a stochastic iterative method that eliminates costly retractions while optimizing on the generalized Stiefel manifold.
• The paper shows that the method converges to a critical point at rates comparable to traditional Riemannian approaches despite using stochastic constraints.
• The paper demonstrates the method’s efficiency in large-scale tasks like CCA, ICA, and GEVP by using a landing strategy based solely on matrix multiplications.

Optimization without Retraction on the Random Generalized Stiefel Manifold for Canonical Correlation Analysis: A Technical Summary

The paper under discussion presents a new approach to optimization problems involving matrices constrained to the generalized Stiefel manifold, characterized by the constraint $X^\top B X = I_p$. This kind of constraint frequently appears in machine learning problems such as Canonical Correlation Analysis (CCA), Independent Component Analysis (ICA), and Generalized Eigenvalue Problems (GEVP). The authors propose a novel stochastic optimization method that eliminates the need for matrix retraction operations, which are computationally expensive, especially when dealing with large-scale matrices.

Theoretical Contributions

1. Stochastic Iterative Method: The paper introduces a stochastic iterative method capable of optimizing over a random estimate of the feasible set without precisely enforcing the constraint at each iteration. This unique feature reduces the per-iteration computational cost compared to traditional Riemannian methods, which typically require full eigenvalue decompositions of the constraint matrix $B$.
2. Convergence Properties: Despite not enforcing the manifold constraint exactly at each iteration, the proposed method guarantees convergence to a critical point of the objective function constrained on the generalized Stiefel manifold. The convergence rates match those of traditional deterministic Riemannian optimization methods.
3. Landing Method: The authors employ a 'landing' strategy that incrementally guides the iterates toward the manifold while allowing stochastic estimates of $B$. This approach leverages matrix multiplications exclusively, avoiding the need for expensive retractions. The landing field is a combination of relative gradient descent directions and a normalization component ensuring convergence.

Numerical Results and Practical Implications

The numerical experiments demonstrate the efficiency and effectiveness of the proposed method in various applications, such as CCA, ICA, and GEVP. The results show that the method outperforms existing techniques, particularly in scenarios where matrix $B$ is large or when only stochastic estimates of $B$ are available.

The practical implications of this research are significant for machine learning applications that deal with large datasets and high-dimensional spaces. The reduced computational cost per iteration makes it feasible to solve complex problems in real-time or near-real-time settings. Furthermore, the stochastic nature of this method aligns well with online learning scenarios, where data is fed sequentially.

Theoretical Implications and Future Directions

From a theoretical standpoint, this paper extends the landscape of optimization techniques on manifolds by incorporating stochastic constraints directly into the framework. This opens several avenues for future research:

• Broadening Applicability: Extending the approach to other types of manifold constraints and different optimization settings, such as non-smooth objectives or non-convex manifolds.
• Improving Convergence Analysis: Further refinement of the convergence analysis could lead to tighter bounds and improved performance guarantees in practical settings.
• Exploration of Other Manifold Structures: While the current work focuses on the generalized Stiefel manifold, exploring similar strategies on other manifold structures could unveil new optimization methods.

In summary, this paper introduces a substantial advancement in the field of manifold optimization. It offers a new methodological perspective by leveraging stochasticity not only in the objective function but also in the constraint, thereby providing a flexible and computationally efficient solution to a class of important machine learning problems. This work sets the foundation for potential developments in both theoretical and applied optimization research.