SingLoRA: Symmetric Low-Rank Adaptation

• SingLoRA is a parameter-efficient fine-tuning technique that replaces dual low-rank matrices with a single symmetric update, addressing instability in classical methods.
• It employs a ramp-up function and symmetric A Aᵀ update, halving the number of trainable parameters while maintaining stable optimization across large-width models.
• Empirical results show SingLoRA outperforms traditional LoRA techniques in language and vision tasks, achieving higher accuracy with reduced computational overhead.

SingLoRA is a parameter-efficient fine-tuning method for large-scale neural networks that modifies the architecture of low-rank adaptation by learning weight updates with a single low-rank matrix and its transpose, rather than the standard product of two distinct low-rank matrices. This design addresses instability and over-parameterization issues that commonly arise in classical Low-Rank Adaptation (LoRA) schemes, providing guaranteed stability in large-width regimes and empirically better accuracy with reduced parameter budgets across language understanding and generative modeling tasks (Bensaïd et al., 8 Jul 2025).

1. Reformulation of Low-Rank Adaptation

Traditional LoRA updates a frozen pretrained weight matrix $W_0 \in \mathbb{R}^{d \times k}$ by the product of two trainable low-rank matrices, $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$, so that $W = W_0 + BA$ (with $r \ll d, k$). Recent findings have shown that mismatched scaling between $A$ and $B$ often leads to unstable optimization: the learning dynamics of each matrix can interfere due to divergent parameter magnitudes, especially as model width grows.

SingLoRA proposes a symmetric update that replaces $BA$ with $A A^\top$, so the adapted model weights take the form:

$$W = W_0 + \frac{\alpha}{r} \cdot u(t) \cdot A A^\top$$

where $B \in \mathbb{R}^{d \times r}$0 is the only trainable matrix, $B \in \mathbb{R}^{d \times r}$1 is a ramp function (typically $B \in \mathbb{R}^{d \times r}$2 over training steps $B \in \mathbb{R}^{d \times r}$3 and ramp period $B \in \mathbb{R}^{d \times r}$4), and $B \in \mathbb{R}^{d \times r}$5 is a scaling hyperparameter. This symmetric construction inherently sidesteps inter-matrix scale conflicts by learning a single parameter matrix.

2. Theoretical Properties and Infinite-Width Analysis

A rigorous analysis in the infinite-width regime demonstrates that SingLoRA’s parameterization ensures stable feature learning by construction. In detail, by adopting scaling rules where the entries of $B \in \mathbb{R}^{d \times r}$6 are initialized (and maintained) at order $B \in \mathbb{R}^{d \times r}$7 (with appropriate learning rates), the symmetric update $B \in \mathbb{R}^{d \times r}$8 preserves output magnitudes at $B \in \mathbb{R}^{d \times r}$9 as the network width $A \in \mathbb{R}^{r \times k}$0.

This eliminates the need for separate learning rate tuning for two matrices and avoids vanishing or exploding gradients—a problem long observed in classical LoRA-based and two-matrix schemes. Consequently, the optimization dynamics remain stable across width scales and throughout training.

3. Methodological Details and Implementation

The update rule in SingLoRA replaces LoRA’s two-matrix structure with a single trainable $A \in \mathbb{R}^{r \times k}$1:

• Initialization: $A \in \mathbb{R}^{r \times k}$2 is initialized with entries $A \in \mathbb{R}^{r \times k}$3. No second matrix is required.
• Updating: At each training step, the update to $A \in \mathbb{R}^{r \times k}$4 is

$A \in \mathbb{R}^{r \times k}$5

where $A \in \mathbb{R}^{r \times k}$6 ramps up linearly from $A \in \mathbb{R}^{r \times k}$7 to $A \in \mathbb{R}^{r \times k}$8 over a warm-up period.

The single symmetric update requires approximately half as many trainable parameters for the same rank $A \in \mathbb{R}^{r \times k}$9 as LoRA and its variants, resulting in reduced memory consumption and potentially smaller communication overhead during distributed fine-tuning. The ramp function $W = W_0 + BA$0 is employed to gradually introduce the low-rank update, further stabilizing early-stage dynamics.

4. Empirical Evaluation and Performance

SingLoRA was validated on multiple tasks across NLP and computer vision:

• Language Understanding: Fine-tuning RoBERTa-base and GPT-2 on GLUE benchmarks (MNLI, QQP, QNLI) showed mean accuracy improvements of approximately $W = W_0 + BA$1 for RoBERTa and $W = W_0 + BA$2 for GPT-2 relative to baseline LoRA, all while using only about $W = W_0 + BA$3 of the trainable parameters.
• Large-Scale LLMs: When applied to LLaMA-7B fine-tuned on MNLI, SingLoRA achieved $W = W_0 + BA$4 accuracy, outperforming LoRA ($W = W_0 + BA$5), LoRA+ ($W = W_0 + BA$6), and DoRA, again at $W = W_0 + BA$7 of the parameter budget.
• Image Generation: In DreamBooth fine-tuning with Stable Diffusion V1.5, SingLoRA improved the DINO similarity score—an image fidelity metric—achieving $W = W_0 + BA$8 compared to $W = W_0 + BA$9 for DoRA and $r \ll d, k$0 for LoRA, and preserved prompt alignment as measured by CLIP text similarity.

These results indicate that SingLoRA matches or surpasses existing parameter-efficient adaptation techniques on common benchmarks in both domains.

5. Applications and Use Cases

SingLoRA is suitable for any scenario where LoRA-style adaptation is beneficial but memory and compute efficiency are critical:

• Parameter-Efficient Fine-Tuning of LLMs: By halving the number of adaptation parameters and stabilizing learning, SingLoRA enables more resource-efficient deployment of large models, especially in multi-task and multi-domain settings.
• Diffusion Models for Image Generation: Its symmetric adaptation proves effective for high-fidelity personalization tasks such as subject-driven generation (DreamBooth), where maintaining subject details and fidelity is challenging for conventional LoRA methods.

A plausible implication is that the symmetric structure of SingLoRA’s update could facilitate new model compression and deployment strategies in constrained or on-device environments.

6. Practical Implications, Limitations, and Outlook

The adoption of Symmetric Low-Rank Adaptation via SingLoRA offers several practical advantages:

• Reduced Parameter Budget: Fewer parameters reduce memory load and may decrease distributed training communication costs.
• Stable Hyperparameter Tuning: Single-matrix adaptation eliminates the need to hand-tune scale or learning rates between two matrices.
• Empirical Robustness: Improved and stable training dynamics translate to better outcomes across tasks without custom schedules or optimization tricks.

Potential limitations include the inherent expressiveness constraints of a symmetric update; scenarios requiring non-symmetric adaptation may still benefit from alternative or composite methods such as DoRA or LoRA+. The empirical results indicate strong performance for $r \ll d, k$1 and $r \ll d, k$2, but further studies on very shallow or specialized architectures are warranted.

Future work may explore hybrid schemes that combine SingLoRA with advanced adaptation modules, ablation of ramp-up strategies, or application to non-standard architectures (multi-modal or recurrent layers). Investigation into theoretical properties beyond the infinite-width regime, as well as hyperparameter sensitivity in resource-constrained deployment, are also promising avenues.

7. Summary Table: SingLoRA vs. Conventional LoRA

Method	Update Parameterization	# Trainable Params	Reported Accuracy (MNLI, LLaMA-7B)	DINO Score (DreamBooth SD)
LoRA	$r \ll d, k$3	$r \ll d, k$4	$r \ll d, k$5	$r \ll d, k$6
LoRA+	Enhanced $r \ll d, k$7 with normalization, etc.	$r \ll d, k$8	$r \ll d, k$9	N/A
DoRA	Decorrelated rank adaptation	Varies	N/A	$A$0
SingLoRA	$A$1	$A$2	$A$3	$A$4

In conclusion, SingLoRA represents a theoretically justified, empirically validated, and methodologically simplified means of parameter-efficient adaptation for large neural networks, exploiting a symmetric single-matrix formulation to guarantee stability while reducing the adaptation parameter count and improving empirical accuracy across NLP and computer vision tasks (Bensaïd et al., 8 Jul 2025).