Manifold-Constrained LLM Adapter Tuning

• Manifold-constrained LLM adapter tuning is a method that optimizes low-parameter adapters by enforcing matrix manifold constraints, such as orthogonality, to boost stability and generalization.
• It employs a three-factor decomposition (W = U S Vᵀ) and advanced Riemannian optimization techniques like MCSD and SPEL to achieve fast, GPU-friendly, single-loop updates.
• By integrating sample weighting and manifold denoising, the approach adaptively fine-tunes models under noisy and domain-shift conditions while reducing memory overhead.

Manifold-constrained LLM adapter tuning refers to methodologies for optimizing low-parameter adapters within LLMs under explicit constraints that require adapter parameters to lie on specified matrix manifolds, typically motivated by stability, orthogonality, and generalization benefits. These approaches integrate advances in Riemannian optimization, norm-constrained linear minimization oracle methods, and manifold-aware sample weighting to enhance adaptation and robustness in both transformers and domain-specialized fine-tuning settings (Yang et al., 29 Jan 2026, Jaberi-Douraki et al., 9 Oct 2025).

1. Manifold Constraints for Adapter Factors

Adapter layers inserted into pretrained LLMs are often re-parameterized to enforce low-rank structure and matrix orthogonality through a three-factor decomposition:

$W = U S V^\top,$

where $U \in \mathbb{R}^{n \times r}$ and $V \in \mathbb{R}^{d \times r}$ must have orthonormal columns, and $S \in \mathbb{R}^{r \times r}$ is diagonal. This constrains $U$ and $V$ to the Stiefel manifolds $$\mathrm{St}(n, r) = \{ X \in \mathbb{R}^{n \times r} : X^\top X = I_r \}$$ and $\mathrm{St}(d, r)$, respectively. The effective search space for adapter optimization is thus a product manifold $$\mathrm{St}(n, r) \times \mathrm{St}(d, r) \times \mathbb{R}^{r \times r}$$, which ensures preservation of key invariances and improves stability when tuning adapters (Yang et al., 29 Jan 2026).

In sample-weighted fine-tuning for domain adaptation, data embeddings are assumed to lie near a smooth, low-dimensional data manifold $\mathcal{M} \subset \mathbb{R}^d$. Quantifying manifold proximity via $$d_{\mathcal{M}}(z) = \inf_{y \in \mathcal{M}} \| z - y \|_2$$, and learning $\mathcal{M}$ via PCA, autoencoders, or diffusion maps, allows adapter weights and loss contributions to be modulated based on geometric relationships to $\mathcal{M}$ (Jaberi-Douraki et al., 9 Oct 2025).

2. Optimization Frameworks: MCSD, SPEL, and LMO Directions

Standard Riemannian gradient methods for manifold-constrained optimization often entail nested iterative schemes for solving tangent-space subproblems. The Manifold Constrained Steepest Descent (MCSD) framework circumvents this by adopting a single-loop update:

• Compute the Euclidean gradient $\nabla_U f(U, S, V)$.
• Project onto the tangent space of the Stiefel manifold to obtain the Riemannian gradient:

$$\nabla_M f(U) = \nabla_U f - U\, \mathrm{sym}(U^\top \nabla_U f).$$

• Identify the steepest descent direction using a linear minimization oracle (LMO) under a spectral norm constraint:

$$d_U = \arg\min_{\|D\|_2 \leq 1} \langle \nabla_M f(U), D \rangle = - \mathrm{msign}(\nabla_M f(U)),$$

where $\mathrm{msign}(X) = X (X^\top X)^{-1/2}$ computes the polar-factor sign matrix.

For the spectral-norm-constrained case, the SPEL (Spectral-Projection Enhanced Learning) specialization implements these operations efficiently via Newton–Schulz iterations (“Polar Express”) to compute $\mathrm{msign}(X)$ without requiring SVD, enabling fast, GPU-friendly updates:

$$X_0 = X / \| X \|_2;\quad X_{k+1} = \frac{1}{2} X_k (3I - X_k^\top X_k),\quad k = 0, \ldots, 7$$

This achieves quadratic convergence to the polar factor (Yang et al., 29 Jan 2026).

3. Retraction and Manifold Projection

After updating $U$ and $V$ by a step $\alpha$ in the ambient space, projection back to the Stiefel manifold is performed via

$$U_+ = \operatorname{Retr}_U(\alpha d_U) = \mathrm{msign}(U + \alpha d_U),$$

which precisely enforces the orthonormal column constraint by mapping to the nearest Stiefel manifold point. Analogous operations are applied for $V$ (Yang et al., 29 Jan 2026).

This ensures that orthogonality is maintained throughout optimization, supporting improved stability and tractability for adapter tuning in LLMs.

4. Sample Weighting and Manifold Denoising via Embedding Geometry

Fine-tuning adapters on mixtures of source and small target data benefits from sample re-weighting schemes grounded in geometric properties of embeddings:

• Similarity-weighted adaptation: Source inputs $x_i$ are re-weighted by $$\omega_i = \exp(-\alpha \cdot \mathrm{dist}_\chi(\mu(x_i), \mu_T))$$, where $\mu$ is the embedding map and $\mu_T$ is the target centroid. $\mathrm{dist}_\chi(\cdot,\cdot)$ includes metrics such as MMD, cosine, or Mahalanobis distances.
• Manifold-based denoising: Off-manifold points receive weights $$\omega_i^\mathrm{clean} = \exp(-\beta \cdot d_{\mathcal{M}}(\mu(x_i)))$$, drastically reducing influence of noisy or outlier samples.

The unified adapter-tuning objective thus incorporates both adaptation and denoising guarantees:

$$L_\text{total}(\theta_a) = \frac{1}{n_s} \sum_{i=1}^{n_s} w^{\text{tot}}_i\, \ell(f_{\theta_0+\theta_a}(x_i), y_i) + \lambda_r \| \theta_a \|_2^2$$

where $$w^{\text{tot}}_i = \omega_i \cdot \omega_i^{\text{clean}}$$ and $\ell$ is the cross-entropy or task loss. Theoretical bounds establish that adaptation fidelity is governed by embedding divergence and sample proximity to $\mathcal{M}$ (Jaberi-Douraki et al., 9 Oct 2025).

5. Hyperparameters and Algorithmic Scheme

In MCSD/SPEL adapter tuning for LLMs (as realized in the StelLA framework):

• Only $U$ and $V$ are updated via the manifold-constrained scheme; $S$, biases, and remaining parameters are optimized with AdamW.
• The base learning rate is $\alpha_\mathrm{base} = 5 \times 10^{-4}$ with linear decay and 500-step warm-up.
• Layerwise scaling applies Muon's rule: for matrix parameters of size $p \times n$, use $$\mathrm{lr} = \alpha_\mathrm{base} \times 0.2 \sqrt{\max(p, n)}$$ for both constrained and unconstrained variables.
• No additional momentum is introduced; MCSD/SPEL utilizes plain updates without heavy-ball momentum for $U$, $V$.

An end-to-end recipe for adapter tuning under manifold constraints is as follows:

1. Initialize $U$, $V$ as random orthonormal matrices on $\mathrm{St}(n, r)$ and $\mathrm{St}(d, r)$. Set $S$ diagonal.
2. At each adapter step: (a) Compute Euclidean gradients; (b) Project into the tangent space for Riemannian gradients; (c) Compute LMO steepest descent directions via $-\mathrm{msign}$; (d) Update in ambient space; (e) Retract via $\mathrm{msign}$ projection. Update other parameters (including $S$ and biases) using AdamW.
3. Adjust learning rates as specified and proceed with linearly decaying schedule (Yang et al., 29 Jan 2026).

6. Empirical Performance and Computational Properties

Comparative results on LLaMA-3-8B and 3.1-8B across eight commonsense-reasoning tasks reveal that SPEL matches or slightly outperforms the original StelLA optimizer while achieving significant memory savings due to its stateless single-loop design.

Optimizer	BoolQ	PIQA	SIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Avg.
StelLA (LLaMA-3–8B)	76.23	89.44	81.68	96.44	88.27	92.49	82.17	87.20	86.74
SPEL (LLaMA-3–8B)	76.25	89.14	81.70	96.18	87.32	91.82	81.80	87.67	86.49
StelLA (LLaMA-3.1–8B)	76.10	89.50	81.41	96.44	87.63	91.93	82.03	87.33	86.55
SPEL (LLaMA-3.1–8B)	76.24	89.94	81.29	96.25	87.03	91.87	81.20	88.00	86.48

SPEL loss curves closely overlap with StelLA across multiple runs. The framework requires approximately 35 GB of additional optimizer state versus approximately 70 GB for AdamW+projection, yielding a twofold reduction in memory requirements. The single-loop design and Newton–Schulz-based msign computations enable full GPU-compatibility and scalability (Yang et al., 29 Jan 2026).

7. Generalization, Domain Adaptation, and Applicability

Manifold-constrained adapter tuning, as demonstrated by MCSD/SPEL and manifold denoising approaches, provides provable guarantees for generalization and robustness in settings subject to domain shift and data noise. In HySim-LLM, theorems quantify the tradeoffs between adaptation, denoising, and sampling error under explicit manifold models and embedding-weighted objectives (Jaberi-Douraki et al., 9 Oct 2025). These techniques extend beyond language modeling to domains with natural low-dimensional manifolds, including structured biomedical data, clinical time series, financial sequences, and omics/genomics, by centering both optimization and sample selection on learned geometric structure.

A plausible implication is that further advances in efficient manifold estimation and projection techniques will continue to improve the scalability and effectiveness of LLM adapter tuning under geometric constraints, supporting broader adaptation across heterogeneous and noisy data regimes.