Weight-Decomposed LoRA (DoRA)

• Weight-Decomposed LoRA (DoRA) is a PEFT method that reparameterizes a fixed pre-trained weight matrix into a trainable magnitude vector and low-rank directional components.
• It decouples magnitude and direction updates to mimic full fine-tuning’s learning behavior while preserving LoRA’s low parameter count and zero additional inference cost.
• Empirical results show DoRA improves training stability and performance across language, vision-language, and instruction-tuning tasks compared to traditional LoRA.

Weight-Decomposed LoRA (DoRA) is a parameter-efficient fine-tuning (PEFT) method that reparameterizes a frozen pre-trained weight matrix into separate magnitude and direction components, then applies low-rank adaptation primarily to the directional component while keeping magnitude explicitly trainable. It was introduced to address the persistent accuracy gap between Low-Rank Adaptation (LoRA) and full fine-tuning (FT), with the stated goal of matching FT-like learning behavior without sacrificing the parameter efficiency and zero additional inference cost associated with LoRA-style adaptation (Liu et al., 2024).

1. Origin, scope, and nomenclature

DoRA emerged from an analysis of why LoRA often lags full fine-tuning even when both are effective. In the original formulation, the central observation is that LoRA entangles changes in weight magnitude and direction inside a single additive low-rank update, whereas FT exhibits more selective behavior across those two components. To study this, the original work decomposed weights into per-column magnitudes and unit-length directions and tracked the evolution of both quantities during training. It reported that LoRA updates show a strong positive correlation between magnitude change and directional change, while FT exhibits a negative correlation; the resulting hypothesis was that LoRA’s inability to disentangle magnitude and direction limits its expressivity relative to FT (Liu et al., 2024).

Within PEFT, DoRA therefore occupies a specific position: it retains the low-rank directional update machinery of LoRA, but augments it with an explicit magnitude parameterization. The original paper states that this improves both the learning capacity and training stability of LoRA while avoiding any additional inference overhead, and reports gains on LLaMA, LLaVA, and VL-BART across commonsense reasoning, visual instruction tuning, and image/video-text understanding tasks (Liu et al., 2024).

The acronym “DoRA” is not unique in the broader PEFT literature. One later paper uses “DoRA” to denote “Dynamic Low-Rank Adaptation,” a method based on decomposing high-rank LoRA layers into structured single-rank components and dynamically pruning them during training (Mao et al., 2024). In the weight-decomposition literature, however, “DoRA” conventionally refers to Weight-Decomposed Low-Rank Adaptation (Liu et al., 2024).

2. Core reparameterization and parameterization

The original DoRA formulation starts from a frozen pre-trained weight $W_0 \in \mathbb{R}^{d \times k}$ and decomposes it into a magnitude vector $m \in \mathbb{R}^{1 \times k}$ and a directional matrix $V \in \mathbb{R}^{d \times k}$ with unit-length columns:

$W_0 = m \cdot \left(\frac{V}{\|V\|_c}\right),$

where $\|\cdot\|_c$ denotes the column-wise $\ell_2$-norm. A low-rank update is then applied to the directional component only:

$$\Delta V = B A,\qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d,k).$$

The adapted weight becomes

$$W' = m \cdot \frac{V + \Delta V}{\|V + \Delta V\|_c}.$$

Equivalently, using the frozen pre-trained direction $W_0$ directly,

$W' = m \cdot \frac{W_0 + B A}{\|W_0 + B A\|_c}.$

In this parameterization, $m \in \mathbb{R}^{1 \times k}$0 is trainable, $m \in \mathbb{R}^{1 \times k}$1 is frozen, and the low-rank factors $m \in \mathbb{R}^{1 \times k}$2 control the directional update (Liu et al., 2024).

Later descriptions preserve the same conceptual split but occasionally write the normalization along rows rather than columns. For example, systems work on high-rank DoRA defines a learnable per-row magnitude vector $m \in \mathbb{R}^{1 \times k}$3 and normalizes each output row of $m \in \mathbb{R}^{1 \times k}$4 by its $m \in \mathbb{R}^{1 \times k}$5-norm, i.e.

$m \in \mathbb{R}^{1 \times k}$6

A later sub-1B study likewise states that DoRA can use a per-row or per-column magnitude vector (Zelenin et al., 23 Mar 2026, Nair et al., 5 Jun 2026). A plausible implication is that the defining idea is the separation of norm-like magnitude from normalized direction, rather than a single mandatory choice of axis.

The trainable parameter budget is only slightly larger than LoRA’s because DoRA adds a single magnitude vector. One later formulation summarizes the count for a single weight $m \in \mathbb{R}^{1 \times k}$7 as $m \in \mathbb{R}^{1 \times k}$8, compared with $m \in \mathbb{R}^{1 \times k}$9 for LoRA and $V \in \mathbb{R}^{d \times k}$0 for full fine-tuning (Qin et al., 2024).

Method	Trainable parameters	Example $V \in \mathbb{R}^{d \times k}$1
FT	$V \in \mathbb{R}^{d \times k}$2	589k
LoRA	$V \in \mathbb{R}^{d \times k}$3	6,144
DoRA	$V \in \mathbb{R}^{d \times k}$4	6,912

For typical Transformer dimensions, the added $V \in \mathbb{R}^{d \times k}$5 parameters are small relative to the low-rank factors. One later paper states that for $V \in \mathbb{R}^{d \times k}$6 and $V \in \mathbb{R}^{d \times k}$7, the extra magnitudes are negligible, specifically $V \in \mathbb{R}^{d \times k}$8 (Wang et al., 2024).

3. Optimization behavior, initialization, and inference

DoRA training keeps the base weight frozen and updates only the magnitude vector and low-rank directional adapter. In the original training recipe, $V \in \mathbb{R}^{d \times k}$9 is initialized from the column norms of $W_0 = m \cdot \left(\frac{V}{\|V\|_c}\right),$0, while $W_0 = m \cdot \left(\frac{V}{\|V\|_c}\right),$1 and $W_0 = m \cdot \left(\frac{V}{\|V\|_c}\right),$2 are initialized so that $W_0 = m \cdot \left(\frac{V}{\|V\|_c}\right),$3 at the start; an example initialization given in the original summary is $W_0 = m \cdot \left(\frac{V}{\|V\|_c}\right),$4 Kaiming uniform and $W_0 = m \cdot \left(\frac{V}{\|V\|_c}\right),$5 (Liu et al., 2024). Other implementations reverse the asymmetry, for example setting $W_0 = m \cdot \left(\frac{V}{\|V\|_c}\right),$6 and $W_0 = m \cdot \left(\frac{V}{\|V\|_c}\right),$7 while still initializing $W_0 = m \cdot \left(\frac{V}{\|V\|_c}\right),$8 from the pretrained norm (Nair et al., 5 Jun 2026).

The optimization behavior differs from vanilla LoRA because the normalization projects gradient flow onto the direction subspace while allowing magnitude to evolve separately. The original paper writes the gradient with respect to the updated directional variable $W_0 = m \cdot \left(\frac{V}{\|V\|_c}\right),$9 as

$\|\cdot\|_c$0

and the gradient with respect to $\|\cdot\|_c$1 as

$\|\cdot\|_c$2

The accompanying interpretation is that DoRA scales and projects the directional gradient away from the current direction while decoupling magnitude updates from direction, which improves conditioning and training stability (Liu et al., 2024).

The inference path remains simple. After training, the learned magnitude and normalized directional update can be merged into a single weight matrix $\|\cdot\|_c$3, and the low-rank factors can be discarded. The original paper explicitly states that $\|\cdot\|_c$4 has the same shape as $\|\cdot\|_c$5, so there is zero additional inference cost or architectural change after merging (Liu et al., 2024). This property is one of the main reasons DoRA is treated as a drop-in replacement for LoRA in PEFT toolchains.

4. Empirical behavior across model families

The initial empirical evidence for DoRA spans language-only, vision-language, and instruction-tuned settings. On eight commonsense reasoning datasets, the original work reports that for LLaMA-7B the average score improves from 74.7% with LoRA to 78.4% with DoRA, and for LLaMA-13B from 80.5% to 81.5%. On LLaMA2-7B and LLaMA3-8B, the reported average gains over LoRA are +2.1% and +4.4%, respectively. For VL-BART, DoRA improves image-text performance from 76.5 to 77.4 and video-text performance from 83.5 to 85.4. For LLaVA-1.5-7B visual instruction tuning, the reported score rises from 66.9 to 67.6. On Alpaca instruction tuning evaluated with MT-Bench and GPT-4 scoring, the reported change is 5.1 to 5.5 on LLaMA-7B and 5.7 to 6.0 on LLaMA2-7B (Liu et al., 2024).

The same study also reports that DoRA outperforms LoRA at all tested ranks $\|\cdot\|_c$6, with especially large gaps at low rank, and that it converges faster and is less sensitive to rank than vanilla LoRA (Liu et al., 2024). This is consistent with the method’s stated goal of better matching FT-like learning patterns under strict parameter budgets.

Later work broadens the empirical record and also makes the picture less uniform. In a sub-1B mathematical reasoning study, LoRA and DoRA are described as performing comparably overall, with task-dependent advantages: DoRA excels in complex reasoning such as GSM8K, whereas LoRA has a slight advantage on pattern-matching-style OrcaMath. The same study further reports that in the “Tiny” regime, defined there as $\|\cdot\|_c$7M parameters, full fine-tuning can underperform even zero-shot baselines, whereas both LoRA and DoRA recover and exceed the baseline (Nair et al., 5 Jun 2026). A plausible implication is that DoRA’s benefits are clearest when PEFT is being used as a stability mechanism as much as an efficiency mechanism.

Small-model evidence points in the same direction. A minBERT case study states that rank-1 decompositions yield negligible performance deficits and that, for very small models, the extra flexibility of DoRA does not translate into higher accuracy; in that study, rank-1 LoRA sufficed (Frees et al., 25 Aug 2025). This directly counters the misconception that DoRA uniformly dominates LoRA across all scales and tasks.

5. Application domains and recurrent misconceptions

Although DoRA was introduced in the context of transformer PEFT, later papers apply the same weight-decomposition principle well beyond standard LLM fine-tuning. In peptide representation learning, PepDoRA adapts the last three Transformer layers of ChemBERTa-77M on a masked language modeling objective and reports downstream improvements over LoRA on membrane permeability, non-fouling, hemolysis, and peptide–protein binding tasks; for example, LoRA top-1 binding accuracy is reported as 60.0% versus 62.1% for DoRA (Wang et al., 2024). In medical imaging, DoRA-C, convDoRA, and CP-DoRA extend the same magnitude–direction decomposition to convolutional kernels in a Unet-based subarachnoid hematoma segmentation setting. That study reports that all LoRA/DoRA variants outperform the best conventional fine-tuning baseline, with DoRA-C at rank 64 reaching a Dice score of $\|\cdot\|_c$8 versus $\|\cdot\|_c$9 for the reported best non-LoRA baseline (Minoccheri et al., 3 Aug 2025).

A common misconception is that DoRA is simply “LoRA with one extra vector.” The additional magnitude vector is indeed the only extra trainable object in the canonical formulation, but the methodological difference is the explicit normalization that constrains the low-rank adapter to directional changes while moving norm control into a separate parameter. Several later papers present this separation as the reason DoRA can better preserve high-confidence pretrained structure or improve conditioning under low-rank constraints (Baqar et al., 14 Feb 2025, Wang et al., 2024).

A second misconception is that DoRA always preserves LoRA’s runtime profile. After merging, the inference path can remain cost-free in the sense stated by the original paper, but training-time and unmerged inference-time overhead can become nontrivial at high rank because the norm computation is expensive. A 2026 systems report states that major frameworks compute the row-wise norm of $\ell_2$0 by materializing the dense product $\ell_2$1, which at $\ell_2$2 and $\ell_2$3 requires about 512 MB of transient working memory in bf16 for a single module (Zelenin et al., 23 Mar 2026). The zero-overhead claim therefore applies to the merged inference form, not to all implementations at all ranks.

6. Variants, generalizations, and systems developments

The most direct structural extension is EDoRA, which keeps the weight-decomposition idea but uses an SVD of the directional component to freeze almost all directions and inserts a small trainable bridge matrix $\ell_2$4. In EDoRA, only the magnitude vector $\ell_2$5 and the bridge $\ell_2$6 receive gradients, yielding $\ell_2$7 trainable parameters per adapted weight, compared with LoRA’s $\ell_2$8 and DoRA’s $\ell_2$9. For a representative Transformer width $$\Delta V = B A,\qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d,k).$$0 and rank $$\Delta V = B A,\qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d,k).$$1, the paper states that this yields $$\Delta V = B A,\qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d,k).$$2 fewer trainable parameters than LoRA and $$\Delta V = B A,\qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d,k).$$3 fewer than DoRA. On six GLUE tasks with RoBERTa-base at rank $$\Delta V = B A,\qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d,k).$$4, it reports 86 K trainable parameters for selected layers, versus 3.2 M for LoRA and 3.3 M for DoRA, while achieving an average GLUE score of 84.48% compared with 83.35% for LoRA and 83.06% for DoRA (Nasiri et al., 21 Jan 2025).

BoRA generalizes DoRA’s one-sided magnitude treatment into a symmetric bi-dimensional decomposition. In that method, both row-wise and column-wise magnitudes are trainable, and the paper states that BoRA surpasses LoRA and DoRA across MT-Bench and commonsense benchmarks; for Llama-2-7b at rank 64, it reports MT-Bench scores of 6.16 for LoRA, 6.38 for DoRA, and 6.76 for BoRA, and commonsense-average accuracies of 77.61, 79.69, and 80.06, respectively (Wang et al., 2024).

DoTA replaces the low-rank matrix update with a Matrix Product Operator decomposition initialized from the pretrained weight. The paper argues that random initialization diverges from the validation loss achieved by FT, and reports that on commonsense reasoning, DoTA outperforms random-initialization methods with fewer parameters; for LLaMA2-7B, it gives a DoTA average of 81.6 with 0.15% of parameters, versus 77.6 with 0.83% for LoRA (Hu et al., 2024). This suggests a broader trend in post-DoRA work toward preserving more of the pretrained structure in the initialization of PEFT adapters.

BiDoRA addresses one of the main criticisms of standard DoRA: the magnitude and direction variables are still optimized synchronously on the same data. BiDoRA formulates the problem as bilevel optimization, with magnitude optimized on a validation split and direction on a training split. Its abstract states that evaluation on fourteen datasets spanning natural language understanding, natural language generation, and token classification reveals that it significantly outperforms DoRA and other PEFT methods (Qin et al., 2024). DoRAN takes a different route, injecting noise into DoRA’s denominator and generating low-rank matrices via auxiliary networks; its abstract likewise states that it consistently outperforms LoRA, DoRA, and other PEFT baselines on vision and language benchmarks (Diep et al., 5 Oct 2025).

At the systems level, “Scaling DoRA” addresses the cost of high-rank normalization. It introduces a factored norm that computes the squared norm of $$\Delta V = B A,\qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d,k).$$5 through base, cross, and Gram terms with $$\Delta V = B A,\qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d,k).$$6 intermediates, and fused Triton kernels that reduce memory traffic by about $$\Delta V = B A,\qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d,k).$$7. Across six 8–32B VLMs at rank $$\Delta V = B A,\qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d,k).$$8, the paper reports that the fused implementation is $$\Delta V = B A,\qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d,k).$$9 faster than Hugging Face PEFT’s DoRA implementation for inference and $$W' = m \cdot \frac{V + \Delta V}{\|V + \Delta V\|_c}.$$0 faster for gradient computation, with up to 7 GB lower peak VRAM, while maintaining final-logit cosine similarity above 0.9999 (Zelenin et al., 23 Mar 2026). These results indicate that DoRA’s long-term viability at high rank depends not only on optimization theory, but also on specialized kernel engineering.