High-Rank Multiplicative PEFT

• High-rank multiplicative PEFT is a model adaptation strategy that uses multiplicative transformations to achieve near full-rank updates with very few trainable parameters.
• Techniques like HyperAdapt, SMoA, and representation editing modulate pretrained weights via scaling and subspace partitioning to enhance expressive capacity.
• Empirical results demonstrate that these methods can rival full fine-tuning, outperforming traditional low-rank approaches under strict parameter constraints.

High-rank multiplicative parameter-efficient fine-tuning (PEFT) encompasses a family of model adaptation strategies that achieve high effective update rank while keeping the number of learnable parameters—relative to full model fine-tuning—extremely low. Unlike traditional low-rank approaches (e.g., LoRA) that inject narrow subspace updates, high-rank multiplicative PEFT methods (including HyperAdapt, SMoA, and representation editing techniques) employ structured or multiplicative transformations to maximize representational capacity per parameter, primarily by clever reweighting and modulation of pretrained model weights or activations (Gurung et al., 23 Sep 2025, Liu et al., 12 Jan 2026, Mao et al., 2024, Wu et al., 2024).

1. Theoretical Principles and Motivation

Conventional PEFT methods (e.g., LoRA) learn additive updates of restricted (typically low) rank, e.g., for weight matrix $W_0 \in \mathbb{R}^{n \times m}$, $\Delta W = UV$ with $U \in \mathbb{R}^{n \times r}$ and $V \in \mathbb{R}^{r \times m}$, leading to $\operatorname{rank}(\Delta W) \leq r$. While computationally efficient and empirically useful, low-rank bottlenecks can underfit downstream tasks demanding full or near-full rank adaptation. High-rank multiplicative approaches instead pursue updates that, despite using far fewer trainable variables than $\mathcal{O}(nm)$, generically induce much higher-rank changes to $W_0$ or its action on hidden representations.

This design is motivated by two observations: (a) many task-aligned directions already exist in pretrained model weights; and (b) reweighting, gating, or modulating these directions can achieve task-specific adaptation without introducing new directions, provided the mechanism has sufficient expressive power (Gurung et al., 23 Sep 2025, Liu et al., 12 Jan 2026, Wu et al., 2024).

2. Multiplicative Update Architectures

HyperAdapt

"HyperAdapt" multiplicatively rescales the rows and columns of a frozen weight matrix $W_0$ via diagonal matrices $A = \operatorname{diag}(a_1, \dots, a_n)$ and $B = \operatorname{diag}(b_1, \dots, b_m)$. The adapted weight is $W_{\text{tuned}} = A W_0 B$, expressible as $W_{\text{tuned}} = (I + D_r)W_0(I + D_c)$. The total number of trainable parameters is $n + m$ for $W_0 \in \mathbb{R}^{n \times m}$, orders of magnitude smaller than either full fine-tuning or LoRA with moderate $r$ (Gurung et al., 23 Sep 2025).

The induced update $\Delta W = W_{\text{tuned}} - W_0$ satisfies $\operatorname{rank}(\Delta W) \leq \min(2r, n, m)$ for $\operatorname{rank}(W_0) = r$. Empirically, for transformer projections with $r \approx \min(n, m)$, $\Delta W$ attains nearly full rank while using only $n + m$ trainable variables.

SMoA

The Structured Modulation Adapter (SMoA) partitions the spectrum of $W_0$ into $K$ disjoint subspaces of approximately equal singular value energy. In each, a local low-rank LoRA-style adapter $(B_k A_k)$ is applied and then multiplicatively modulated (Hadamard product) with a frozen spectral mask derived from $W_0$’s SVD. Formally, for subspace $k$:

$$\widehat{\Delta W}_k = (B_k A_k) \odot \widetilde\Sigma_k,$$

and the full update is block-diagonal:

$$\Delta W = \mathrm{block\text{-}diag}(\widehat{\Delta W}_1, ..., \widehat{\Delta W}_K).$$

With carefully balanced subspaces, SMoA achieves an effective update rank up to $r d$ (where $d$ is block size, $r$ per-subspace rank), greatly exceeding LoRA’s limit (Liu et al., 12 Jan 2026).

Representation Editing (RED) and Related Approaches

Representation editing (RED) applies element-wise scaling and bias to internal activations, $h_2 = s_\ell \odot h_1 + b_\ell$. Each scaling $s_\ell$ and bias $b_\ell$ is a length-$d$ vector, with $d$ the feature dimension. The diagonal scaling matrix $\operatorname{diag}(s_\ell)$ is generically full-rank, so this approach effects high-rank per-layer transformations with a minimal parameter overhead ($2d$ per layer). No task-specific rank hyperparameter is required; capacity matches the feature dimension by construction (Wu et al., 2024).

3. Empirical and Theoretical Performance

High-rank multiplicative PEFT methods routinely match or nearly match the performance of full model fine-tuning and outperform classic low-rank approaches under severe parameter budget constraints:

Method	Params (RoBERTa-Large)	GLUE Avg.	Params (Qwen-2.5-7B)	Math Reasoning	Commonsense Reasoning
Full FT	355M	88.2	—	—	—
LoRA (r=32)	0.8M	87.8	1.05M	87.1	86.8
HyperAdapt	0.2M	86.4	0.03M	86.9	85.5
RED	0.02M	84.3	0.26M (Llama-2-7B)	—	—
SMoA (K=2)	Comparable to LoRA	Superior	Comparable	Superior	Superior

HyperAdapt’s normalized “usable rank fraction,” computed as the fraction of singular values above $10^{-2}$ in $\Delta W$, reaches $[0.9, 1.0]$ (average $\approx 0.97$) across transformer modules—far above classic LoRA, whose normalized rank typically falls below $0.1$ (Gurung et al., 23 Sep 2025). SMoA’s learned $\Delta W$ achieves higher empirical rank than LoRA, MoRA, and other variants; its parameter efficiency and block-diagonal update structure provide improved expressivity and downstream accuracy, particularly on complex tasks (Liu et al., 12 Jan 2026).

4. Methodological Variants and Dynamic Rank Schemes

Dynamic rank allocation extends high-rank PEFT by redistributing parameter budgets across layers and tasks:

• DoRA expresses LoRA updates as sums of $r'$ rank-1 components, dynamically prunes them by estimated importance (Frobenius norm-based), and allocates more capacity to “key” modules (e.g., Transformer queries, keys, output projections). The “Dimensional Equilibrium Modulator” penalty regularizes to prevent instability from pruned, ‘spiky’ components. DoRA achieves higher performance than LoRA and AdaLoRA within identical parameter budgets, and often matches full fine-tuning using as little as 0.34M trainable parameters, allocating resources where most impactful (Mao et al., 2024).

A plausible implication is that dynamic rank schemes synergize with multiplicative structures to further maximize update rank and adaptation capacity under strict parameter and memory constraints.

5. Training, Regularization, and Implementation

Training of high-rank multiplicative PEFT typically involves freezing $W_0$ and optimizing only the scaling, modulation, or adapter parameters. Commonalities include:

• Initialization: Multiplicative/diagonal scales and biases are initialized to unity or zero (preserving $W_0$ or activations at iteration zero).
• Optimizer: AdamW with learning rates in $1\times10^{-3}$ to $6\times10^{-2}$ range for scales and modulations, and minor weight decay for regularization.
• Rank/parameter efficiency: These methods require dramatically fewer parameters per adapted module (e.g., $n+m$ for HyperAdapt, $2d$ per layer for RED, adjustable via $K$ in SMoA).
• Inference cost: Multiplicative methods (e.g., HyperAdapt, SMoA) can precompute adapted weights, incurring no extra inference latency relative to pretraining or LoRA (Gurung et al., 23 Sep 2025, Liu et al., 12 Jan 2026).
• Task and model compatibility: While primarily demonstrated on large pretrained LLMs (e.g., RoBERTa, Llama, Qwen, Phi), extension to other backbones remains an active research direction (Gurung et al., 23 Sep 2025).

6. Limitations and Open Questions

Several limitations and open questions exist for high-rank multiplicative PEFT:

• Reliance on pretrained $W_0$: Methods such as HyperAdapt require that $W_0$ is already well-trained and approximately full-rank; they are ineffective on randomly initialized or low-rank base weights (Gurung et al., 23 Sep 2025).
• Scope of demonstrated generality: The majority of empirical results have been established on transformer-based NLP models, with application to vision and generative models an open direction (Gurung et al., 23 Sep 2025).
• Row vs. column scaling roles: The theoretical and practical interplay between row versus column scalings and their task-dependent importance remains incompletely understood (Gurung et al., 23 Sep 2025).
• Over-fragmentation risk: In SMoA, excessive subdivision into subspaces (large $K$) may over-fragment the spectrum and degrade adaptation performance on some tasks (Liu et al., 12 Jan 2026).

7. Comparative Analysis with Low-Rank PEFT

High-rank multiplicative PEFT represents a paradigm shift from the additive, low-rank update principle of classical PEFT. While LoRA imposes a fixed low-rank bottleneck ($\operatorname{rank} \leq r$ for $\Delta W$), high-rank multiplicative approaches generically approach or saturate the full rank of $W_0$ (or of the feature dimension) with minimal parameter cost. Empirical head-to-head comparisons consistently show competitive or superior accuracy from high-rank multiplicative methods under strict memory and parameter budgets (Gurung et al., 23 Sep 2025, Liu et al., 12 Jan 2026, Mao et al., 2024, Wu et al., 2024).

Notable real-world best practices:

• Use HyperAdapt when extreme parameter efficiency and full-rank updates are desired with negligible inference and memory cost.
• Employ SMoA for heterogeneous, complex reasoning tasks demanding high global update rank; tune subspace number $K$ and per-block rank accordingly.
• Dynamic allocation and pruning (e.g., DoRA) can further focus limited resources and stabilize training.

In summary, high-rank multiplicative PEFT enables rich and expressive adaptation of large pretrained models with minimal overhead, driven by structured and theoretically grounded transformations that maximize update rank per parameter (Gurung et al., 23 Sep 2025, Liu et al., 12 Jan 2026, Mao et al., 2024, Wu et al., 2024).