A Smooth Locally Exact Penalty Method for Optimization Problems over Generalized Stiefel Manifolds

Abstract: In this paper, we consider a class of optimization problems constrained to the generalized Stiefel manifold. Such problems are fundamental to a wide range of real-world applications, including generalized canonical correlation analysis, linear discriminant analysis, and electronic structure calculations. Existing works mainly focuses on cases where the generalized orthogonality constraint is induced by a symmetric positive definite matrix M, a setting where the geometry essentially reduces to that of the standard Stiefel manifold. However, many practical scenarios involve a singular M, which introduces significant analytical and computational challenges. Therefore, we propose a Smooth Locally Exact Penalty model (SLEP) and establish its equivalence to the original problem in the aspect of stationary points under a finitly large penalty parameter. This penalty model admits the direct application of various unconstrained optimization techniques, with convergence guarantees inherited from established results. Compared to Riemannian optimization approaches, our proposed penalty mode eliminates the need for retractions and vector transports, hence significantly reducing per-iteration computational costs. Extensive numerical experiments validate our theoretical results and demonstrate the effectiveness and practical potential of the proposed penalty model SLEP.