Low-Rank MLP Parameterization Methods

• Low-Rank MLP parameterization is a technique that decomposes weight matrices into lower-dimensional forms, significantly reducing parameters while preserving performance.
• Factorized, regularized, and hypernetwork-based approaches enable efficient fine-tuning and adaptation, with methods like LoRA and PoLAR demonstrating competitive results.
• Empirical benchmarks show that these low-rank adaptations match or outperform dense models in tasks such as language modeling, computer vision, and commonsense reasoning.

A low-rank MLP parameterization denotes any approach for representing the weight matrices of multilayer perceptrons (MLPs) in a form that constrains or induces the matrix rank to be much smaller than the ambient dimension. This paradigm dramatically reduces parameter and compute complexity, facilitates parameter-efficient adaptation, and can exploit emergent properties of network training dynamics. The field includes factorized, regularized, reparameterized, and group-structured methods, and recent advances demonstrate that low-rank parameterizations match or surpass dense (full-rank) baselines in a wide range of adaptation, fine-tuning, and even pretraining contexts.

1. Factorized Low-Rank Parameterizations

The classical approach factorizes a weight matrix $W \in \mathbb{R}^{m \times n}$ into a product of two lower-dimensional matrices: $W = U V^\top$ with $$U \in \mathbb{R}^{m \times r},\ V \in \mathbb{R}^{n \times r}$$ and $r \ll \min(m, n)$. This reduces parameter count from $mn$ to $r(m+n)$, significantly compressing large MLPs and reducing their inference footprint (Barone, 2016). The basic forward mapping for one layer is

$y = \phi(U V^\top x + b).$

Adding a diagonal term $D$ (low-rank-plus-diagonal) improves expressivity in aggressively compressed regimes: $W = U V^\top + D$, with $D=\mathrm{diag}(d)$. Skip-connection (“passthrough”) architectures further decouple expressivity from rank by ensuring the network state can propagate information even through bottlenecked weights.

Rank selection is task-dependent: for example, $W = U V^\top$0 or $W = U V^\top$1 suffices for language modeling or synthetic sequence modeling; extreme settings ($W = U V^\top$2) require the diagonal enhancement. Universal approximation is preserved if $W = U V^\top$3, and, via stacking and passthrough, even tighter bottlenecks can approximate arbitrary mappings to precision $W = U V^\top$4 with polynomial depth (Barone, 2016).

2. Low-Rank Parameterization in Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT) for large pre-trained models often uses low-rank adaptation, with LoRA (“Low-Rank Adaptation”) serving as paradigm. Instead of training the full $W = U V^\top$5, LoRA learns an update in a low-rank subspace: $W = U V^\top$6 The adapted matrix is $W = U V^\top$7, updating only $W = U V^\top$8 parameters per adapted layer. This approach preserves memory and computation efficiency and is widely adopted in transformer architectures for both attention and MLP sublayers (Bihany et al., 9 Jun 2025).

Several modern extensions of this basic framework address expressivity, optimization, and statistical efficiency:

• Multiplicative Low-Rank (LoRMA): Moves beyond the additive $W = U V^\top$9 to multiplicative forms $$U \in \mathbb{R}^{m \times r},\ V \in \mathbb{R}^{n \times r}$$0, where $$U \in \mathbb{R}^{m \times r},\ V \in \mathbb{R}^{n \times r}$$1 (with $$U \in \mathbb{R}^{m \times r},\ V \in \mathbb{R}^{n \times r}$$2 low-rank), exploring a strictly richer set of transforms via full-rank parameter inflation (identity or permutation-based), leading to improved empirical rank and faster convergence at equivalent parameter budgets (Bihany et al., 9 Jun 2025).
• Polar Decomposition (PoLAR): Employs a polar-style factorization $$U \in \mathbb{R}^{m \times r},\ V \in \mathbb{R}^{n \times r}$$3 with $$U \in \mathbb{R}^{m \times r},\ V \in \mathbb{R}^{n \times r}$$4 constrained to be column-orthogonal (on the Stiefel manifold) and $$U \in \mathbb{R}^{m \times r},\ V \in \mathbb{R}^{n \times r}$$5 unconstrained, which provably enforces high “stable rank” and ameliorates collapse to a single dominant direction, thus improving utilization of the nominal rank $$U \in \mathbb{R}^{m \times r},\ V \in \mathbb{R}^{n \times r}$$6 and accelerating convergence via Riemannian optimization (Lion et al., 3 Jun 2025).
• Bayesian and Monte Carlo Methods: Techniques such as MonteCLoRA endow the low-rank factors $$U \in \mathbb{R}^{m \times r},\ V \in \mathbb{R}^{n \times r}$$7 with mixture-of-Gaussian hierarchies and/or Bayesian priors, allowing sampling and marginalization, which stabilizes fine-tuning, reduces estimator variance, and enhances robustness to hyperparameter settings (Sengupta et al., 2024).
• Hypernetwork & Overparameterized Variants: Approaches like RepLoRA and OP-LoRA use small MLPs (“hypernetworks”) to generate $$U \in \mathbb{R}^{m \times r},\ V \in \mathbb{R}^{n \times r}$$8 and $$U \in \mathbb{R}^{m \times r},\ V \in \mathbb{R}^{n \times r}$$9 from codes or embeddings, exploiting overparameterization for implicit adaptation of learning rates and momentum, and achieving both statistical efficiency and improved optimization dynamics (Truong et al., 5 Feb 2025, Teterwak et al., 2024).

3. Theoretical Foundations for Low-Rank MLPs

Recent theoretical work has established that standard MLPs trained by gradient descent on smooth activations undergo weight updates concentrated in invariant low-dimensional subspaces (Xu et al., 5 Feb 2026). For a two-layer MLP $r \ll \min(m, n)$0 with output dimension $r \ll \min(m, n)$1, one finds:

• The Jacobian of the loss $r \ll \min(m, n)$2 has rank at most $r \ll \min(m, n)$3 at initialization (for smooth $r \ll \min(m, n)$4 and small $r \ll \min(m, n)$5).
• Throughout training, the weight dynamics for $r \ll \min(m, n)$6 remain within a $r \ll \min(m, n)$7-dimensional subspace, with the remaining directions updated at only $r \ll \min(m, n)$8 scale.
• This leads to the parameterization

$r \ll \min(m, n)$9

where $mn$0 is $mn$1 and captures almost all effective learning. Initializing $mn$2 according to the dominant singular vectors of the initial gradient ensures that training with this low-rank $mn$3 matches full-rank MLP performance on classification tasks, provided output dimension $mn$4 is small.

Empirical validations on datasets like Fashion-MNIST and CIFAR-10 confirm that low-rank parameterizations with $mn$5 match the accuracy of dense models, if properly initialized in the correct subspace (Xu et al., 5 Feb 2026).

4. Advanced Regularization and Rank Control

Explicit low-rank regularization presents a route for continuous rank-induction, with methods exploiting smooth surrogates for the rank function. The Quadratic Reweighted Rank Regularizer (Q3R) (Ghosh et al., 6 Nov 2025) replaces non-differentiable rank penalties with a smoothed log-determinant,

$mn$6

where $mn$7 switches between quadratic and (logarithmic) penalties depending on the singular value magnitude. Training alternates truncated SVD with IRLS-inspired quadratic majorization, ensuring that the solution remains within an explicit rank budget. After training, matrices are truncated for highly efficient inference. Q3R achieves reductions in parameter count >50% with minimal accuracy drop on ViT and transformer models.

Second-order optimization with differentiable bilinear parameterization, as formalized in VarPro/LM methods, provides a smooth surrogate to classical nuclear norm or hard-rank penalties. The bilinear form $mn$8 admits explicit quadratic regularizers on $mn$9, is highly amenable to second-order optimization, and yields rapid convergence even for ill-conditioned models (Örnhag et al., 2018).

5. Structured, Joint, and Analytical Parameterizations

Expressivity and efficiency can be further enhanced by structured factorizations and analytical post-training low-rank reductions:

• Joint Tensor-Train (TT) Parameterization: By jointly generating multiple low-rank matrices (e.g., for up- and down-projections) using a shared TT-core network, one can enforce correlated adaptation, yielding both improved parameter efficiency and optimization dynamics. The TensorGuide framework shows that such joint TT parameterizations outperform both classical and per-matrix TT decompositions, as measured by faster convergence and improved accuracy (Qi et al., 19 Jun 2025).
• Analytical CUR-based Selection (A³): Instead of factorization, the A³ approach analytically selects the best $r(m+n)$0 neuron dimensions in the MLP, forming CUR-type masks and reducing the hidden width directly. This post-training procedure replaces bottleneck sublayers with smaller ones selected via a data-informed heuristic. This yields reduced memory and compute without inference overhead and outperforms typical SVD-like layerwise compression (Wong et al., 19 May 2025).

6. Optimization, Sample Efficiency, and Empirical Benchmarks

Low-rank parameterizations can raise optimization challenges, such as ill-conditioning, sensitivity to initialization, and slow convergence. Overparameterization (hypernetwork-based) and structured reparameterizations mitigate these issues by providing adaptive effective learning rates and improved search spaces (Teterwak et al., 2024, Truong et al., 5 Feb 2025). Bayesian and Monte Carlo methods stabilize the optimization trajectory and reduce variance in model outputs (Sengupta et al., 2024).

Empirically, low-rank MLP parameterizations have demonstrated strong or state-of-the-art results in:

• Natural language understanding (GLUE): LoRMA and PoLAR match or marginally surpass dense or LoRA-adapted baselines with order-of-magnitude fewer parameters (Bihany et al., 9 Jun 2025, Lion et al., 3 Jun 2025).
• Commonsense reasoning, vision-language multi-task benchmarks, and mathematical reasoning: Polarization, multiplicative updates, and joint-structured parameterizations consistently yield empirical gains (Lion et al., 3 Jun 2025, Qi et al., 19 Jun 2025).
• Compression and acceleration: Analytical and Q3R-based methods achieve and maintain superior performance at parameter reduction rates >60%, with negligible loss in accuracy (Wong et al., 19 May 2025, Ghosh et al., 6 Nov 2025).
• Downstream utility: Overparameterized and Bayesian variants enhance robustness to optimizer and batch-size choices, stabilize training, and accelerate convergence.

7. Comparison, Trade-offs, and Open Directions

A summary of recently proposed parameterizations:

Method	Param. Count	Expressivity	Optimization
Classical LoRA	$r(m+n)$1	Additive low-rank	Fast, easy, but limited stable rank
LoRMA	$r(m+n)$2	Multiplicative, full-rank via inflation	Matched or better than LoRA
PoLAR	$r(m+n)$3	Enforced stable-rank, polar decomposition	Riemannian optimization
Q3R	$r(m+n)$4	Explicit rank via regularization	IRLS, moderate overhead
RepLoRA/OP-LoRA	$r(m+n)$5 + small MLP	Overparam., adaptive	Hypernetwork acceleration
TensorGuide	$r(m+n)$6	Joint/structured, TT	NTK-theoretically faster
A³	--	Analytical selection	Post-training, no inference overhead

Recent work shows that naive low-rank parameterization risks under-utilizing the subspace (low stable rank), slow convergence, or collapse to a single direction. Structured approaches (PoLAR, TensorGuide), multiplicative parameterizations (LoRMA), and hypernetwork-based reparametrization (RepLoRA, OP-LoRA) avoid these pitfalls, with empirical gains in accuracy, convergence speed, robustness, and parameter efficiency. For tasks with small output dimension $r(m+n)$7, emergent low-rank training dynamics of MLPs mean that parameterizing and training only the “large-movement” 2$r(m+n)$8-dimensional subspace is near-optimal (Xu et al., 5 Feb 2026).

A plausible implication is that task-specific low-rank parameterizations, augmented with geometric, probabilistic, or hypernetwork structure, will remain central in the scalable adaptation and compression of deep models. Future work may explore further integration of low-rank priors at pretraining, learnable rank-scheduling strategies, and mixed-method hybridizations.

---

References:

• (Barone, 2016) Low-rank passthrough neural networks
• (Örnhag et al., 2018) Bilinear Parameterization For Differentiable Rank-Regularization
• (Sengupta et al., 2024) Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation
• (Teterwak et al., 2024) OP-LoRA: The Blessing of Dimensionality
• (Truong et al., 5 Feb 2025) RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts
• (Wong et al., 19 May 2025) A3: an Analytical Low-Rank Approximation Framework for Attention
• (Lion et al., 3 Jun 2025) PoLAR: Polar-Decomposed Low-Rank Adapter Representation
• (Bihany et al., 9 Jun 2025) LoRMA: Low-Rank Multiplicative Adaptation for LLMs
• (Qi et al., 19 Jun 2025) Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation
• (Ghosh et al., 6 Nov 2025) Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
• (Xu et al., 5 Feb 2026) Emergent Low-Rank Training Dynamics in MLPs with Smooth Activations