DoRA: Weight-Decomposed Low-Rank Adaptation

• DoRA is a parameter-efficient tuning technique that decomposes pretrained weights into distinct magnitude and directional components.
• It adapts only the directional part via low-rank updates while explicitly learning magnitude, better aligning with full fine-tuning behavior than LoRA.
• Empirical results across language and multimodal models demonstrate DoRA’s performance gains and maintenance of zero additional inference overhead.

DoRA, short for Weight-Decomposed Low-Rank Adaptation, is a parameter-efficient fine-tuning (PEFT) method that decomposes a pretrained weight into magnitude and direction, learns the magnitude explicitly, and applies low-rank adaptation only to the directional component. It was introduced to reduce the accuracy gap between LoRA and full fine-tuning (FT) while preserving LoRA’s defining deployment property: no additional inference overhead (Liu et al., 2024).

1. Position within parameter-efficient fine-tuning

DoRA was proposed in the context of widespread use of LoRA and related PEFT methods for adapting large pretrained models. In standard LoRA, a frozen weight $W_0$ is modified by a low-rank additive update,

$W' = W_0 + \Delta W = W_0 + BA,$

with $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$, and $r \ll \min(d,k)$. This design is attractive because it avoids additional inference costs after merging, but the original DoRA paper argues that the remaining gap to FT is not explained solely by the number of trainable parameters (Liu et al., 2024).

A central claim of the method is that LoRA’s update pattern itself is different from FT’s. The paper identifies a deeper limitation than simple rank deficiency: LoRA tends to couple magnitude and direction updates together, whereas FT exhibits a different geometry of adaptation. This reframes the problem from one of parameter count alone to one of update structure. In that sense, DoRA is not merely a larger LoRA; it is a reparameterization intended to better match the way FT changes weights.

This motivation also addresses a common misconception in PEFT discussions. The paper explicitly contests the view that LoRA underperforms FT only because it uses too few trainable parameters. Instead, the authors argue that the more important issue is the mismatch between LoRA’s low-rank additive update geometry and FT’s observed behavior.

2. Weight decomposition analysis of full fine-tuning and LoRA

The analytical foundation of DoRA is a decomposition of a weight matrix $W \in \mathbb{R}^{d \times k}$ into a magnitude vector and a directional matrix: $W = m \frac{V}{\|V\|_c},$ where $m \in \mathbb{R}^{1 \times k}$ is the magnitude vector, $V \in \mathbb{R}^{d \times k}$ is the directional matrix, and $\|\cdot\|_c$ is the column-wise norm. Under this decomposition, each column of $W' = W_0 + \Delta W = W_0 + BA,$0 is a unit vector, while $W' = W_0 + \Delta W = W_0 + BA,$1 controls scale (Liu et al., 2024).

Using this factorization, the paper compares pretrained weights, FT weights, and merged LoRA weights through two summary quantities: $W' = W_0 + \Delta W = W_0 + BA,$2, which measures average magnitude difference, and $W' = W_0 + \Delta W = W_0 + BA,$3, which measures average directional difference. The empirical finding is that FT often shows a negative relationship between $W' = W_0 + \Delta W = W_0 + BA,$4 and $W' = W_0 + \Delta W = W_0 + BA,$5, whereas LoRA shows a positive relationship. The paper interprets this as evidence that LoRA does not reproduce FT’s more flexible tradeoff between scale and direction changes.

The authors describe this discrepancy as a learning capacity gap. In their interpretation, LoRA is less able to realize patterns such as “large directional change with small magnitude change” or the converse, while FT can do so more naturally. This suggests that a PEFT method which decouples these two components may better approximate FT’s effective hypothesis class.

3. Formal definition of DoRA

DoRA retains the low-rank efficiency of LoRA, but reparameterizes the adapted weight so that magnitude and direction are updated separately. Initialization begins from the pretrained weight: $W' = W_0 + \Delta W = W_0 + BA,$6 The base direction is then frozen, and LoRA is used to model only the directional update $W' = W_0 + \Delta W = W_0 + BA,$7. The resulting adapted weight is

$W' = W_0 + \Delta W = W_0 + BA,$8

Here, $W' = W_0 + \Delta W = W_0 + BA,$9 is trainable, and $B \in \mathbb{R}^{d \times r}$0 are trainable low-rank matrices (Liu et al., 2024).

Operationally, the method can be summarized in three steps. First, it normalizes the pretrained weight into direction. Second, it learns magnitude separately. Third, it uses LoRA only for directional adaptation. This separation is the defining distinction from standard LoRA, which must encode both kinds of change inside a single additive low-rank update.

The paper argues that this decomposition improves the effective learning problem in two ways. Magnitude can now be adjusted explicitly instead of being implicitly entangled with direction, and the low-rank update can be focused on directional adaptation alone. This is intended to make DoRA’s adaptation behavior closer to FT without sacrificing PEFT efficiency.

4. Optimization properties, training behavior, and deployment

Beyond representational flexibility, the paper claims that DoRA improves training stability. Its gradient analysis shows that the gradient for the directional component is scaled by $B \in \mathbb{R}^{d \times r}$1 and projected away from the current weight direction, while the magnitude receives a separate gradient signal. The authors argue that this improves conditioning and makes optimization more stable, in a manner described as similar in spirit to weight normalization (Liu et al., 2024).

A practical issue is that the normalization term $B \in \mathbb{R}^{d \times r}$2 can increase backpropagation memory use. The paper therefore proposes a training optimization in which this norm is treated as a constant and detached from the gradient graph. Reported memory savings are about 24.4% on LLaMA-7B and about 12.4% on VL-BART, with little or no accuracy loss.

At inference time, DoRA preserves the principal systems advantage of LoRA. The learned components can be merged back into the pretrained weight before deployment, so there is no additional inference overhead. This point is central to the method’s positioning: it is meant to increase adaptation quality without converting PEFT into a heavier inference-time architecture.

These properties make DoRA a methodologically hybrid design. It borrows LoRA’s deployment model, but replaces LoRA’s additive parameterization with a normalized magnitude-direction decomposition intended to align better with FT’s empirically observed update geometry.

5. Empirical performance across language and multimodal models

The original evaluation covers both language and multimodal backbones and reports that DoRA consistently outperforms LoRA (Liu et al., 2024). On commonsense reasoning, the paper evaluates LLaMA-7B, LLaMA-13B, LLaMA2-7B, and LLaMA3-8B over eight benchmarks: BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA. Reported average gains over LoRA are +3.7 on LLaMA-7B, +1.0 on LLaMA-13B, +2.9 on LLaMA2-7B, and +4.4 on LLaMA3-8B. The paper also reports that a reduced-rank variant, $B \in \mathbb{R}^{d \times r}$3, still beats LoRA even with half the rank.

On VL-BART for image-text and video-text understanding, DoRA outperforms LoRA on VQA v2, GQA, NLVR$B \in \mathbb{R}^{d \times r}$4, COCO Caption, TVQA, How2QA, TVC, and YC2C. The summary figures reported in the paper are roughly +0.9 on image-text benchmarks and +1.9 on video-text benchmarks. The paper further states that DoRA gets close to or reaches FT-level performance in this setting.

On LLaVA-1.5-7B for visual instruction tuning, the benchmark suite includes VQA v2, GQA, VisWiz, SQA, TextVQA, POPE, and MMBench. DoRA improves over LoRA by about 0.7 average points and also improves over FT in this setting. The paper notes that LoRA already slightly beats FT on LLaVA here, likely due to FT overfitting, and DoRA still improves further.

The method is also presented as compatible with other low-rank schemes. Combined with VeRA, it yields DoVA, which outperforms VeRA on MT-Bench with Alpaca instruction tuning and also improves over LoRA. Additional experiments show that DoRA is better than LoRA across ranks $B \in \mathbb{R}^{d \times r}$5, is especially strong when rank is small, and remains robust when training data is reduced.

6. Later extensions, systems scaling, and acronym reuse

Subsequent work extended the original formulation in two principal directions. DoRAN introduces a learnable positive offset $B \in \mathbb{R}^{d \times r}$6 in the denominator,

$B \in \mathbb{R}^{d \times r}$7

and replaces static low-rank matrices with auxiliary networks that generate them dynamically. The stated goals are to stabilize the strict normalization used in DoRA and to improve sample efficiency through parameter coupling across layers and heads. The paper reports that DoRAN consistently outperforms LoRA, DoRA, and other PEFT baselines on both vision and language tasks, and it gives a theoretical result for the shared/noise-perturbed formulation of

$B \in \mathbb{R}^{d \times r}$8

in its statistical analysis (Diep et al., 5 Oct 2025).

A separate systems-focused line of work addresses the computational cost of high-rank DoRA. “Scaling DoRA” observes that standard implementations materialize the dense product $B \in \mathbb{R}^{d \times r}$9 to compute $A \in \mathbb{R}^{r \times k}$0, which becomes expensive at large $A \in \mathbb{R}^{r \times k}$1 and rank. It proposes a factored norm that replaces the dense intermediate with $A \in \mathbb{R}^{r \times k}$2 intermediates, and fused Triton kernels that reduce memory traffic by about 4×. Across six 8B–32B vision-LLMs at $A \in \mathbb{R}^{r \times k}$3 in bf16, the fused implementation is reported as 1.5–2.0× faster than Hugging Face PEFT’s DoRA for inference, 1.5–1.9× faster for gradient computation, and uses up to 7 GB less peak VRAM, while maintaining final-logit cosine similarity above 0.9999 (Zelenin et al., 23 Mar 2026).

The acronym DoRA has also been reused in a different PEFT paper, Dynamic Low-Rank Adaptation, which decomposes a high-rank LoRA layer into structured single-rank components and prunes them dynamically during training. That work targets adaptive rank-budget allocation rather than magnitude-direction decomposition, and reports strong results on GLUE, SQuAD, and XSum under fixed storage budgets (Mao et al., 2024). In current PEFT usage, however, the unqualified term DoRA most often denotes the Weight-Decomposed Low-Rank Adaptation formulation introduced in 2024 (Liu et al., 2024).