Subspace-Constrained Training Method

• Subspace-Constrained Training Method is a machine learning strategy that restricts model parameters and gradients to a low-dimensional, structured subspace to enforce desired properties.
• It employs manifold optimization techniques, such as tangent space projections and retraction methods, to maintain orthogonality and control partitioning of high-dimensional data.
• This method is applied to feature learning, domain adaptation, and robust deep network optimization, offering theoretical convergence guarantees and improved sample efficiency.

A subspace-constrained training method is a machine learning and optimization strategy in which the model parameters, gradients, or feature representations are explicitly or implicitly restricted to evolve within a structured, often low-dimensional subspace of the ambient high-dimensional space. Such approaches arise in contexts including manifold optimization, robust learning, memory-efficient deep learning, subspace clustering, and structured regularization. The purpose is typically to encode desired properties (such as orthogonality, sparsity, or partitioning), inject prior knowledge, enforce interpretability, enhance generalization, or reduce computational burden.

1. Mathematical Foundations and Manifold Formulation

Subspace constraints arise naturally in problems where the solution must satisfy algebraic or geometric structure, such as orthogonality or partitioning. The partitioned subspace (PS) manifold (Giguere et al., 2017) generalizes classical Grassmannian and Stiefel manifolds. If an $n$-dimensional ambient space must be partitioned into $m$ mutually orthogonal subspaces of user-defined sizes $k_1, \ldots, k_m$ (with $k = \sum_{i=1}^m k_i \leq n$), the constraint set is:

$$\text{PS}(n; k_1, \ldots, k_m) = \mathcal{O}_n / \mathcal{E}_{PS},$$

with $\mathcal{O}_n$ the orthogonal group and the equivalence set comprising block-diagonal orthogonal transformations:

$$\mathcal{E}_{PS} = \mathrm{diag}(\mathcal{O}_{k_1}, \ldots, \mathcal{O}_{k_m}, \mathcal{O}_{n-k}).$$

Each point on the PS manifold represents a matrix $Q$ whose columns span the concatenation of orthogonal subspaces.

Optimization then proceeds using Riemannian geometry: the tangent space at $Q$ consists of matrices $Q A$ for skew-symmetric $A$, projected to discard “vertical directions” (i.e., those representing basis changes within partitions). Gradients are projected onto this tangent space and updated using retraction (e.g., via QR decomposition). This guarantees that learned representations remain within the required subspace structure.

2. Optimization Algorithms and Tangent Space Updates

The subspace-constrained optimization framework is not exclusive to partitioned subspaces. In many problems, constraints manifest as manifold optimization or projection operations:

• In the PS framework (Giguere et al., 2017), a general update $Z$ is projected:

$$\pi_{T_Q \mathrm{PS}}(Z) = Q \cdot \text{skew}(Q^\top Z) \quad \text{(with block-diagonal blocks zeroed out)}.$$

• For subspace trust region methods (Dudar et al., 2018), optimization is restricted to adaptive, low-dimensional subspaces (spanned by gradient and history), with curvature exploited by solving a quadratic minimization:

$$\min_{\|\alpha\| \leq \epsilon} Q(\alpha) = -r^\top \alpha + \frac{1}{2} \alpha^\top B \alpha$$

and further restricted to positive curvature directions to avoid saddle points.

• In randomized subspace projection methods (Nozawa et al., 2023), updates are constructed by projecting gradients into randomly chosen subspaces or the subspace spanned by active constraints, allowing for larger step sizes and faster convergence in high-dimensional constrained settings.
• In Riemannian meta-optimization (Yu et al., 25 Jan 2025), gradient adaptation occurs by learning per-coordinate updates in the row and column subspaces, with only the diagonals of the gradient covariance matrices passed through small LSTM modules, yielding

$\widehat{G} = R \cdot G \cdot C,$

with $R$ and $C$ block-diagonal and LSTM-adapted.

3. Expressivity, Partitioning, and Custom Constraints

A principal strength of the partitioned subspace manifold (Giguere et al., 2017) is its flexibility in enforcing complex orthogonality and grouping constraints that go beyond what is possible with classical Stiefel or Grassmann manifolds.

• If $m=1$ and $k_1 = k$, the manifold reduces to the Grassmannian.
• If $m = k$ and $k_i=1$, it reduces to the Stiefel manifold.
• For $m > 1$, arbitrary grouping and mutually orthogonal subspaces can be learned and regularized independently.

This enables, for example, feature representations where individual partitions are tailored for shared or domain-specific structure (multi-dataset analysis), or class-discriminative representations (domain adaptation; see Section 4 below). Further, the tangent space structure ensures that gradient updates cannot “mix” bases between different groups, preserving strict invariance under chosen group actions.

Constrained subspace approximation (Bhaskara et al., 29 Apr 2025) generalizes these constraints beyond orthogonality, supporting explicit/implicit constraint sets $S$ on the projection matrix $P$. The “coreset-guess-solve” paradigm reduces the high-dimensional, non-convex original problem to a sequence of convex regression problems via projection cost-preserving coresets and coefficient enumeration, yielding $(1+\varepsilon)$-approximation guarantees even under fairness, partition, or nonnegativity requirements.

4. Applications: Feature Learning, Transfer, Clustering, and Robust Methods

Applications of subspace-constrained training methods are numerous:

• Multiple Dataset Analysis (MD-PCA): The PS manifold provides a natural parameterization where each dataset or feature subset is assigned a specific partition. Experiments on Office+Caltech10 showed extracted features align with common and dataset-specific structure (Giguere et al., 2017).
• Domain Adaptation: By assigning per-class partitions (or background partitions as needed), class-discriminative transfer subspaces can be learned, with joint optimization of reconstruction and discriminative objectives. This led to robust performance in transfer learning tasks, often surpassing classical approaches (e.g., GFK, Subspace Alignment) in Naive Bayes settings (Giguere et al., 2017).
• Constrained Subspace Clustering: In sparse subspace clustering with side-information, affinity matrices are constructed via weighted $\ell_1$ self-expressiveness where weights encode must-link/cannot-link relationships. Representation learning is performed in the constraint-informed subspace, and spectral clustering with additional constraints yields improved error rates, especially with limited supervision (Li et al., 2018). Theoretical bounds on clustering accuracy are derived by linking estimation to the Rand index.
• Low-Dimensional Manifold Learning: Quadratic matrix factorization methods with subspace constraints can simultaneously recover tangent and normal spaces, enabling high-fidelity manifold denoising and embedding; alternating minimization is guaranteed to converge to stationary points under mild strong convexity and curvature conditions (Zhai et al., 7 Nov 2024).

5. Practical Implementation and Performance Guarantees

Empirical studies demonstrate strong numerical stability and competitive or superior error when employing subspace-constrained approaches:

• On synthetic and real data, the PS manifold, SCPAST (sparse constrained projection approximation subspace tracking (Belomestny et al., 2018)), and robust Tyler’s estimator with subspace constraint (Lerman et al., 27 Mar 2024) provide robustness to outliers, accuracy with limited inlier fractions, and improved sample efficiency over unconstrained or naively regularized baselines.
• In neural network training, subspace methods (trust region, subspace-momentum, online subspace descent) accelerate convergence, mitigate saddle point stalling, and reduce memory requirements, as illustrated in AdamSNSM (Nguyen et al., 11 Nov 2024), SubTrack++ (Rajabi et al., 3 Feb 2025), and OSD (Liang et al., 23 Aug 2024).
• Randomized subspace optimization (Chen et al., 11 Feb 2025) further reduces memory/communication in LLM training by solving low-dimensional projected subproblems at each iteration (with efficient sampling of projection matrices), while offering convergence guarantees matching vanilla Adam and comparable perplexity to full-dimensional approaches.

Key theoretical results include:

• Linear or quadratic rates of subspace estimation error with SNR (2003.11738).
• Non-asymptotic error bounds in online subspace tracking (Belomestny et al., 2018).
• $(1+\varepsilon)$ or $\varepsilon$-additive approximation for constrained $k$-means, partition-constrained subspace approximation, and projected NMF with running time exponential only in $k$ for small $k$ (Bhaskara et al., 29 Apr 2025).
• r-linear convergence and robustness under weak inlier–outlier models for subspace-constrained Tyler’s estimator (Lerman et al., 27 Mar 2024).

6. Implications for Advanced Optimization and Future Directions

Subspace-constrained training methods offer a principled route to designing learning algorithms that natively encode structural and task-driven constraints through geometry, covering domains from clustering and manifold learning to robust statistics and scalable deep network optimization. These methods facilitate:

• Modular objective decomposition with per-partition or per-group regularization.
• Automatic enforcement of mutual exclusivity, fairness, or other combinatorial properties without ad hoc penalties.
• Sharp theoretical guarantees, including rates, stability, and robustness in non-i.i.d. or adversarial settings.

Emerging variants—such as hybrid adaptive/projected optimizers, dynamic online PCA/SVD subspace estimation, and subspace-aware meta-optimization—promise further improvements in efficiency, robustness, and memory scaling, especially for extreme-scale foundation models and resource-constrained scenarios.

Continued research is likely to focus on extending these tools to broader classes of constraints (beyond orthogonality), adaptive and online subspace tracking, integration with distributed and federated settings, and automated selection/scheduling of subspace updates for heterogeneous model components.