Parameterized Stiefel MLP (P-SMLP)

• P-SMLP is an orthogonality-constrained neural architecture that employs manifold-aware retractions to preserve strict orthogonality and robust gradient flow.
• It uses both SVD and QR-based retractions to project intermediate outputs onto the Stiefel manifold, ensuring consistent performance on product manifold optimization problems.
• Empirical results demonstrate its ability to achieve machine-precision accuracy in PGIEP tasks, highlighting its potential for stable, deep neural optimization.

The parameterized Stiefel multilayer perceptron (P-SMLP) is an orthogonality-constrained neural architecture designed for optimization tasks on product manifolds such as the parameterized generalized inverse eigenvalue problem (PGIEP). It employs hard constraints via manifold-aware retractions, enabling both efficient end-to-end training and guaranteed orthogonality throughout the entire network (Zhang et al., 25 Jan 2026). The approach unifies advances in orthogonal matrix parameterizations and Stiefel manifold optimization, allowing neural networks to attain perfect dynamical isometry and robust gradient flow even in very deep settings (Massucco et al., 4 Aug 2025).

1. Product Manifold Optimization: PGIEP Formulation

The PGIEP considers affine matrix pencils $A(c) = A_0 + \sum_{i=1}^n c_i A_i$ and $B(c) = B_0 + \sum_{i=1}^n c_i B_i$ with a prescribed target spectrum $\lambda \in \mathbb{R}^n \cup \{\infty\}$. The solution seeks orthogonal matrices $Q, Z \in O(n)$ such that

$Q^\top A(c) Z = S, \quad Q^\top B(c) Z = T$

where $S$, $T$ are upper-triangular. Constraints on the diagonal and strict upper-triangular structure are enforced via a composite loss function: $$\text{Loss}(Q,Z,c; \lambda) = \frac{1}{2} \| \Lambda \odot (Q^\top B(c) Z) - I \odot (Q^\top A(c) Z) \|_F^2 + \frac{1}{2} \| P \odot (Q^\top A(c) Z) \|_F^2 + \frac{1}{2} \| P \odot (Q^\top B(c) Z) \|_F^2$$ with $$\Lambda = \operatorname{diag}(\lambda_1,\dots,\lambda_n)$$ and $P$ the strict-lower-triangular mask (Zhang et al., 25 Jan 2026).

2. P-SMLP Network Architecture

A P-SMLP generates orthogonal matrix outputs through a multilayer pipeline:

• Input: Initial orthogonal $X \in \mathbb{R}^{n \times n}$.
• Hidden Spine: $L$ fully connected layers, $H_0 = X$, $$H_\ell = \Phi(\omega_{\ell-1} H_{\ell-1} + b_{\ell-1})$$, for $\ell = 1,\dots,L$, with activations $\Phi$ (typically ReLU) and trainable parameters $\theta = \{\omega_\ell, b_\ell\}$.
• Final Layer: Outputs $f(X) \in \mathbb{R}^{n \times n}$, $c \in \mathbb{R}^n$ by a linear split.
• Stiefel Layer ("Editor’s term"): Orthonormalization operator $\Psi$ projects $f(X)$ onto $O(n)$ via SVD-based or QR-based retraction. This yields $Q, Z \in O(n)$ for use in the loss (Zhang et al., 25 Jan 2026).

3. Manifold-Constrained Parameterization and Hard Constraints

Orthogonalization of outputs is realized by projection/retraction:

• SVD-based retraction: Given $M = f(X)$, compute $U\Sigma V^\top = \text{svd}(M)$ and set $Q = UV^\top$;
• QR-based retraction: Thin QR, $M = Q'R$, with $$Q = Q' \operatorname{diag}(\operatorname{sign}\operatorname{diag} R)$$. Both approaches are smooth almost everywhere and guarantee $Q^\top Q = I$ regardless of parameter drift. This "hard constraint" means no alternating projection or penalty terms are needed, simplifying the algorithmic workflow (Zhang et al., 25 Jan 2026).

4. Training and Gradient Continuity

The training procedure is end-to-end and leverages differentiable retractions embedded in automatic differentiation frameworks (e.g., PyTorch):

• Forward pass: Generate $Q, Z$ via $\Psi$ after the MLP spine and compute the composite loss as above.
• Backward pass: Gradients $\partial \text{Loss}/\partial \theta$ are propagated automatically through the retraction layers, ensuring manifold constraints are respected.
• Optimization: Standard Adam or SGD may be used; deterministic convergence is supported by gradient Lipschitz continuity of the objective, with the bound

$$\| \nabla F(c,Q,Z) - \nabla F(\hat{c},\tilde{Q},\tilde{Z}) \|_F \leq L ( \|c - \hat{c}\|_2 + \|Q - \tilde{Q}\|_F + \|Z - \tilde{Z}\|_F )$$

for some $L > 0$ (Zhang et al., 25 Jan 2026).

5. Practical Implementation and Computational Complexity

Each forward pass involves orthogonalization atop an MLP spine:

• Complexity: Each epoch requires an SVD or QR on an $n \times n$ matrix, scaling as $O(n^3)$ per iteration. For moderate $n$ (e.g., $n \lesssim 100$), this cost is minor relative to other network operations.
• Initialization: Orthogonal matrices may be drawn by random normal followed by QR decomposition to ensure $O(n)$ membership.
• Regularization: No separate penalty or alternating step is needed due to the hard constraint; the architecture is robust to numerical drift.
• Stability: Empirically, SVD-based $\Psi$ often exhibits better convergence, and activation choice (tanh vs. ReLU) can influence stability for larger $n$ (Zhang et al., 25 Jan 2026).

6. Applications and Empirical Performance

P-SMLP has demonstrated robust convergence and accuracy across various PGIEP problem instances:

• Asymmetric and symmetric PGIEP: Achieved machine-precision or $<10^{-4}$ residuals for network sizes from $n=2$ up to $n=40$.
• Multiple eigenvalues/singular $B$: Consistently recovers parameters $c$ and associated matrices, even in ill-conditioned cases.
• End-to-end differentiation: P-SMLP does not require alternating eigen-solves and remains differentiable throughout (Zhang et al., 25 Jan 2026). This approach generalizes to other tasks requiring coupled Euclidean and orthogonal (Stiefel) factors, including orthogonal dictionary learning, Riemannian VAEs, and control (Zhang et al., 25 Jan 2026).

7. Theoretical Context and Connection to Orthogonal Networks

The P-SMLP framework is structurally related to recent advances in orthogonal neural architectures:

• Enforcement of hard orthogonality ensures perfect dynamical isometry and avoids vanishing/exploding gradients, as established in the theory of networks with orthogonal Jacobians (Massucco et al., 4 Aug 2025).
• Manifold-aware retractions (Cayley, exponential, Householder, SVD/QR) are compatible with modern nonconvex optimization theory and empirically match the trainability of best-in-class residual networks (Massucco et al., 4 Aug 2025).
• P-SMLP can be viewed as an instantiation of Stiefel-layered MLPs, whose strict constraints yield provable convergence and strong spectral guarantees.

In summary, parameterized Stiefel multilayer perceptrons offer a principled, scalable architecture for neural optimization over product manifolds with rigorous orthogonality constraints, supporting both theoretical guarantees and empirical robustness across challenging matrix inverse eigenvalue tasks (Zhang et al., 25 Jan 2026).