Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction

Abstract: The memory wall remains the primary bottleneck for training LLMs on consumer hardware. We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) V^T, where the full dense matrix is never materialized during training or inference. Gradients flow through the compact spectral factors via standard backpropagation, and U, V are retracted to the Stiefel manifold via QR decomposition after each optimizer step. SCT achieves up to 199x memory reduction per MLP layer at rank 32, enabling full training steps of 70B-parameter architectures on a Steam Deck handheld (7.2 GB peak memory vs. 1,245 GB for dense FP32 training with Adam). Rank-sweep experiments on SmolLM2-1.7B (ranks 32-256, 2000 steps, NVIDIA A100) show that all tested ranks converge to the same loss floor (~4.2-4.5), identifying the learning rate schedule -- not MLP rank -- as the primary bottleneck. Rank 128 emerges as the efficiency sweet spot at 11.7x MLP compression with the lowest perplexity. GPU memory drops 46% at rank 32 while training throughput doubles.

• The paper introduces a method that replaces dense weight matrices with permanent truncated SVD factors while enforcing orthonormality via Stiefel QR retraction.
• It achieves up to 199× memory reduction per MLP layer, enabling full training of 70B-scale models on commodity devices.
• Experimental results reveal a favorable rank–quality tradeoff and show that optimizer configuration, rather than spectral rank, dominates convergence performance.

Spectral Compact Training: Permanent Truncated SVD for LLM Pre-Training

Introduction and Motivation

Spectral Compact Training (SCT) introduces an architecture-agnostic weight parameterization that addresses the substantial memory demands of LLM training. Unlike established post-training SVD-based compression or side-adapter approaches such as LoRA, SCT replaces each dense weight matrix with a persistent truncated SVD factorization, $W = U\,\mathrm{diag}(s)\,V^\top$, held for the entire training process and inference. Critically, gradients are propagated through these spectral factors, and the orthogonality of $U, V$ is explicitly enforced via Stiefel manifold retraction using a QR decomposition after every optimizer update.

SCT enables a memory reduction of up to 199$\times$ per MLP layer at low ranks on architectures as large as 70B parameters, reducing total training memory from $\sim$1.2TB to under 8GB—achieving full training on commodity hardware. This approach offers a direct path toward democratizing both training and research on massive LLMs.

Figure 1: SCT reduces 70B-architecture training memory requirements by 172$\times$ compared to conventional dense training.

Methodological Foundations

Permanent Spectral Parameterization

Instead of training over parameter matrices of shape $m \times n$, SCT stores only the truncated SVD factors: $U \in \mathbb{R}^{m \times k}$, $s \in \mathbb{R}^k$, $V \in \mathbb{R}^{n \times k}$. This design never materializes $W$ itself during forward, backward, or inference computations, retaining strict low-rankness throughout the model's lifecycle. The forward pass operates via three compact linear operations:

• $U, V$0 ($U, V$1)
• $U, V$2 ($U, V$3)
• $U, V$4 ($U, V$5)

This reduces both parameter count and compute footprint.

Backpropagation and Stiefel Retraction

Standard automatic differentiation computes gradients directly with respect to $U, V$6. Since these factors are required to remain orthonormal, after each optimizer update (e.g., AdamW), $U, V$7 and $U, V$8 are retracted to the Stiefel manifold using QR decomposition, enforcing exact orthonormality rather than soft penalties. This is structurally distinct from regularization-based approaches and directly preserves spectral integrity for subsequent updates.

Compared to LoRA or related efficient fine-tuning paradigms, SCT does not retain or ever construct the original dense weights but operates entirely in the spectral domain. In contrast to approaches such as GaLore, which project only optimizer states, SCT does not generate or store any dense gradients or states throughout.

Experimental Results

Full-Scale Training Feasibility

SCT was validated on a canonical 70B transformer model (architectural match to LLaMA-3-70B) with additive attention. At rank 32, complete forward, backward, optimizer, and QR retraction steps are possible on a consumer-class device in under 8 GB of memory. The methodology eliminates the historical requirement for massive compute clusters to experiment with full-scale architectures.

Rank–Quality Tradeoff

Compression–quality dynamics were further analyzed via a rank sweep (32, 64, 128, 256) on the SmolLM2-1.7B model fine-tuned on Alpaca. All SCT ranks converge to a similar loss floor (4.2–4.5), sharply contrasting with the dense baseline (1.29), but demonstrating that within SCT, spectral rank is not the dominant convergence bottleneck.

Figure 2: Loss convergence is consistent across SCT ranks, all attaining a similar minimum loss floor.

Notably, rank 128 was identified as Pareto-optimal, simultaneously achieving an 11.7$U, V$9 compression, peak GPU memory reduction from 35.5 GB to 20 GB, and best perplexity among SCT settings (65.6). Even the largest spectral rank (256) achieves a 5.9$\times$0 compression ratio and reduces VRAM demand by 40%.

Figure 3: Left: Pareto frontier illustrating optimal tradeoff at rank 128. Right: Even at high ranks, SCT achieves significant VRAM reductions.

Analysis of Convergence Gap

The uniformity in loss floor across ranks, contrasted with a >3-point gap to the dense baseline, implicates non-optimal learning rate schedules rather than SVD rank constraints as the limiting factor. The dense baseline was trained at $\times$1, with all SCT configurations at $\times$2. This learning rate appears to be suboptimal, particularly for higher ranks that better preserve pretrained structure; further, attention layers were left dense and trained at the elevated SCT rate, emphasizing the need for per-component and perhaps per-rank scheduling.

Fine-Tuning Gradient Integrity

On smaller models, SCT fine-tuning with orthonormal retraction was validated to retain proper gradient flow and learning dynamics, with a minor degradation (1.38$\times$3 perplexity) even when compression benefits are limited.

Theoretical and Practical Implications

SCT establishes low-rank, strict-orthonormal weight parameterization not as a post-hoc or fine-tuning operation but as a first-class training scheme for LLMs. The approach demonstrates that substantial savings in both parameter count and runtime memory can be realized without catastrophic loss in training viability. Importantly, rank hyperparameters and retraction frequency can be decoupled and explored independently, and SCT is compatible with other architectural modifications.

Practical implications include the democratization of training and debugging of massive models on affordable hardware platforms, enabling broader experimental access and iteration. From a theoretical perspective, SCT aligns with trends in optimization-on-manifolds for deep learning, embedding hard constraints into parameter updates as opposed to soft regularization or separate compression pipelines.

Limitations and Directions for Future Work

The principal outstanding challenge is the convergence gap to full-density training; evidence thus far attributes this primarily to suboptimal optimizer settings rather than inherent limitations of the low-rank manifold parameterization. Automated, dynamic, and per-component learning rate scheduling schemes appear necessary for closing this gap.

Retracting via QR decomposition incurs costs that may be non-trivial—accounting for up to 50% of step time in the 70B setting—though alternatives such as Cayley retraction are available and merit empirical assessment.

Currently, only MLP layers are spectrally parameterized. Extending SCT to attention projections is straightforward but may introduce subtle alignment and quality issues, particularly with respect to multi-head structure and compositionality.

Performance benefits diminish for architectures with small hidden dimensions, as spectral ranks close to the full dimension are needed for non-destructive compression.

Demonstration of full-scale, long-horizon pre-training (as opposed to downstream fine-tuning benchmarks) remains an open milestone.

Conclusion

Spectral Compact Training offers a powerful, memory-efficient route to pre-training and fine-tuning of LLMs by leveraging permanent truncated SVD factorization with Stiefel manifold retraction. The method enables training of 70B-scale architectures within 8 GB of memory, with favorable Pareto-optimal tradeoffs between rank, memory, and perplexity. Contrary to conventional intuition, the choice of spectral rank within a reasonable range does not set the convergence floor—optimizer configuration currently dominates. SCT lays the groundwork for manifold-based model training and immediate practical reductions in hardware barriers for building large neural language systems.