Orthogonal Directions Constrained Gradient Method

• ODCGM is a first-order optimization method that minimizes functions on smooth manifolds using orthogonal projections.
• It alternates between feasibility and tangential descent updates to enforce nonlinear equality constraints without explicit retraction.
• ODCGM achieves near-optimal convergence rates and efficiency in high-dimensional settings, notably on the Stiefel manifold.

The Orthogonal Directions Constrained Gradient Method (ODCGM) is a class of first-order optimization algorithms for minimizing functions over smooth manifolds, especially those defined by nonlinear equality constraints. ODCGM leverages orthogonal projection techniques to ensure that iterative updates are aligned with the tangent space of the constraint manifold, thereby enabling efficient optimization without explicit feasibility enforcement. This framework has been applied broadly, including optimization over the Stiefel manifold, providing convergence rates that are provably near-optimal for both deterministic and stochastic settings (Schechtman et al., 2023).

1. Mathematical Foundations and Problem Statement

ODCGM addresses constrained optimization problems of the form: $$\min_{x \in M} f(x) \quad \text{where} \quad M = \{ x \in \mathbb{R}^n \mid h(x) = 0 \}$$ with $h: \mathbb{R}^n \to \mathbb{R}^{n_h}$ continuously differentiable and $\nabla h(x)$ having full rank $n_h$ locally. The framework admits both general nonlinear equality constraints and classical matrix manifolds such as the Stiefel manifold $$M = \{ X \in \mathbb{R}^{p \times q} \mid X^\top X = I_q \}$$ (Schechtman et al., 2023).

At each point $x$, the method utilizes:

• The “constraint-violation” gradient: $\nabla H(x) = \nabla h(x) h(x)$.
• The tangent space: $V(x) = \ker(\nabla h(x)^\top)$.
• The orthogonal projector onto the tangent space:

$$P_V = I_n - G (G^\top G)^{-1} G^\top, \quad G = \nabla h(x)$$

yielding the projected gradient $\nabla_V f(x) = P_V \nabla f(x)$.

2. Algorithmic Structure and Update Rule

The ODCGM update scheme iteratively alternates between driving the iterate toward feasibility and performing a tangential descent:

• Compute the feasibility direction: $h: \mathbb{R}^n \to \mathbb{R}^{n_h}$0, with $h: \mathbb{R}^n \to \mathbb{R}^{n_h}$1 ensuring positive definiteness.
• Compute the tangential direction: $h: \mathbb{R}^n \to \mathbb{R}^{n_h}$2.
• Update:

$h: \mathbb{R}^n \to \mathbb{R}^{n_h}$3

No retractions or explicit feasibility corrections are performed: iterates may be infeasible but are systematically pulled toward the manifold $h: \mathbb{R}^n \to \mathbb{R}^{n_h}$4. A stochastic version substitutes the true gradient with unbiased noisy estimates (Schechtman et al., 2023).

For implementation on matrix manifolds like the Stiefel manifold, tangent projections reduce to solving the Sylvester equation or a canonical form $h: \mathbb{R}^n \to \mathbb{R}^{n_h}$5 for $h: \mathbb{R}^n \to \mathbb{R}^{n_h}$6.

3. Theoretical Properties: Convergence and Oracle Complexity

Under standard regularity assumptions (Lipschitz gradient on $h: \mathbb{R}^n \to \mathbb{R}^{n_h}$7 and $h: \mathbb{R}^n \to \mathbb{R}^{n_h}$8, compact level sets), the deterministic ODCGM achieves: $h: \mathbb{R}^n \to \mathbb{R}^{n_h}$9 Thus, an $\nabla h(x)$0-stationary and $\nabla h(x)$1-feasible point is reached in $\nabla h(x)$2 gradient evaluations.

In the stochastic regime (unbiased gradient estimators, variance $\nabla h(x)$3, stepsize $\nabla h(x)$4), the expected violation decays as $\nabla h(x)$5, resulting in $\nabla h(x)$6-precision after $\nabla h(x)$7 gradient evaluations—matching lower complexity bounds for nonconvex, first-order methods (Schechtman et al., 2023).

4. Connections to Related Methods and the “Landing” Algorithm

On the Stiefel manifold, ODCGM with suitable metric choices recovers the discrete “landing” algorithm of Ablin–Peyré (2022). Specifically, choosing $\nabla h(x)$8, $\nabla h(x)$9 in the ODCGM update yields: $n_h$0 The improved analysis in ODCGM establishes $n_h$1 and $n_h$2 rates for the landing algorithm, which surpass the previous suboptimal $n_h$3 bound and demonstrate convergence to the manifold (Schechtman et al., 2023).

5. Numerical Performance and Implementation Highlights

ODCGM and its variants demonstrate superior empirical performance in high-dimensional and ill-conditioned settings:

• On Procrustes problems ($n_h$4), ODCGM and geometry-aware variants outperform explicit Riemannian gradient methods by orders of magnitude in both objective reduction and constraint satisfaction across $n_h$5 and $n_h$6.
• For highly discretized mechanics problems (e.g., hanging chain with $n_h$7 up to $n_h$8), ODCGM converges to feasible solutions in $n_h$9 iterations, remaining computationally efficient ($$M = \{ X \in \mathbb{R}^{p \times q} \mid X^\top X = I_q \}$$0s per update for $$M = \{ X \in \mathbb{R}^{p \times q} \mid X^\top X = I_q \}$$1).
• No feasibility projection or singular value decomposition is needed due to the infeasible, vector-space-projection-based structure (Schechtman et al., 2023).

6. Extensions and Generalizations

ODCGM encapsulates and extends several classical optimization paradigms:

• On the Stiefel manifold, it unifies Euclidean/ambient and intrinsic (Riemannian) gradient schemes.
• The same projection-based philosophy is compatible with a wide range of nonlinear constraint geometries, provided only that the constraint Jacobian maintains full rank locally.
• The method's avoidance of expensive manifold retractions and its reliance solely on orthogonal projections renders it applicable in large-scale nonconvex settings.

7. Practical Limitations and Future Directions

While theoretically well-founded, ODCGM does not preserve feasibility at every step; iterates are only asymptotically feasible, which may pose issues in applications requiring strict constraint satisfaction throughout optimization. The full efficacy for constraints beyond smooth equalities (e.g., inequalities, nonsmooth or rank-deficient constraints) is not addressed by the current analysis. Extensions to these broader constraint classes and deeper exploration of acceleration and higher-order variants remain active areas of research (Schechtman et al., 2023).

---

Key Reference:

• "Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifold" (Schechtman et al., 2023)