Flat-LoRA: Full-Parameter Fine-Tuning

• Flat-LoRA is a PEFT method that adapts low-rank updates in flat regions of the full parameter space, ensuring robust generalization.
• It transfers full-space adversarial perturbations to the low-rank subspace using a Bayesian objective and pseudo-inverse, avoiding cross-term interference.
• The method improves performance across diverse tasks while maintaining near-LoRA computational efficiency and a low parameter count.

Flat-LoRA is a Parameter-Efficient Fine-Tuning (PEFT) method for large pre-trained models that seeks a low-rank adaptation located in a flat region of the full parameter space, rather than merely a flat region inside the constrained LoRA optimization space. In the original formulation, the motivating observation is that a solution that appears flat in the loss landscape of the LoRA space may still exhibit sharp directions in the full parameter space, which can compromise generalization; Flat-LoRA addresses this without adopting the standard sharpness-aware minimization procedure with its associated computation and memory overheads, instead using a Bayesian expectation loss objective and a refined random perturbation generation strategy (Li et al., 2024). Subsequent work gives a more explicit sharpness-aware formulation, shows how perturbations in the full parameter space can be transferred to the low-rank subspace, and introduces EFlat-LoRA as an efficient variant with near-LoRA optimization cost (Deng et al., 1 Aug 2025).

1. Problem setting and motivating intuition

Flat-LoRA arises from a specific limitation of standard LoRA. LoRA constrains training to a low-dimensional subspace of updates, so that only low-rank matrices are optimized while the pre-trained backbone remains frozen. This makes fine-tuning efficient in memory and computation, but it also means that the geometry seen during optimization is only a projection of the geometry of the full model. The central claim behind Flat-LoRA is that flatness in the LoRA subspace does not guarantee flatness in the full parameter space (Li et al., 2024).

The later theoretical treatment makes the point more sharply in terms of sharpness-aware optimization. A long line of theory and empirical evidence links flatter minima to better generalization, and SAM improves generalization by optimizing a robust objective based on adversarial weight perturbations. In LoRA, however, directly applying SAM to the two low-rank factors can create a mismatch between the perturbation neighborhood in the low-rank parameterization and the intended neighborhood in the full weight matrix. The proposed Flat-LoRA framework is therefore motivated by the need to seek flat minima for LoRA in a way aligned with the full parameter space, rather than by perturbing LoRA factors independently (Deng et al., 1 Aug 2025).

This framing also clarifies what Flat-LoRA is not. It is not merely a regularized LoRA variant that prefers small updates; rather, it is a procedure for making low-rank adaptation responsive to full-space sharpness. This suggests that its contribution is geometric as much as parametric: the method changes the effective neighborhood in which robustness is optimized.

2. LoRA parameterization and the mismatch with naïve SAM

The formal setup starts from a frozen base weight matrix $W_0 \in \mathbb{R}^{n \times m}$. LoRA introduces a rank-$r$ update with $r \ll \min(n,m)$ through two trainable factors,

$$\Delta W = sBA, \qquad W = W_0 + \Delta W, \qquad y = (W_0 + sBA)x,$$

where $B \in \mathbb{R}^{n \times r}$ is the up-projection, $A \in \mathbb{R}^{r \times m}$ is the down-projection, and $s>0$ is the LoRA scaling hyperparameter. The reported initialization is standard: $A$ is Kaiming-initialized, $B$ is zero-initialized, $W_0$ is frozen, and only $r$0 and $r$1 are trainable (Deng et al., 1 Aug 2025).

SAM optimizes a robust objective of the form

$r$2

with the first-order perturbation approximation

$r$3

Its sharpness proxy is the perturbed loss increase $r$4, and the standard implementation requires two forward/backward passes (Deng et al., 1 Aug 2025).

The difficulty appears when this is applied directly to the LoRA factors. A naïve LoRA-only robust objective can be written as

$r$5

The crucial issue is the cross-term $r$6. According to the theoretical analysis, this term introduces interference between the two low-rank subspaces and makes the inner maximization inconsistent with the full-parameter inner maximization around $r$7 (Deng et al., 1 Aug 2025).

That interference is the main reason Flat-LoRA does not simply add SAM on top of LoRA. The method instead attempts to preserve the semantics of full-space adversarial perturbation while remaining inside the low-rank parameterization.

3. Perturbation transfer from the full parameter space

The core theoretical contribution is a perturbation-transfer construction. Rather than perturbing both LoRA factors independently, the method starts from the full-space robust objective

$r$8

and then transfers the full-space perturbation $r$9 to a perturbation on only one low-rank factor, typically $r \ll \min(n,m)$0, while fixing $r \ll \min(n,m)$1 so that $r \ll \min(n,m)$2 (Deng et al., 1 Aug 2025).

Under a first-order approximation, the transfer is implemented with the Moore–Penrose pseudo-inverse of $r \ll \min(n,m)$3:

$r \ll \min(n,m)$4

The stated intuition is that

$r \ll \min(n,m)$5

which yields a first-order loss match and eliminates the problematic $r \ll \min(n,m)$6 cross-term by construction (Deng et al., 1 Aug 2025).

Because LoRA training does not explicitly store or directly compute $r \ll \min(n,m)$7, the full-space gradient is estimated from LoRA gradients in two first-order consistent ways:

$r \ll \min(n,m)$8

and

$r \ll \min(n,m)$9

For stability, the two are averaged:

$$\Delta W = sBA, \qquad W = W_0 + \Delta W, \qquad y = (W_0 + sBA)x,$$0

A SAM-style full-parameter perturbation is then defined in Frobenius norm as

$$\Delta W = sBA, \qquad W = W_0 + \Delta W, \qquad y = (W_0 + sBA)x,$$1

and transferred to the LoRA $$\Delta W = sBA, \qquad W = W_0 + \Delta W, \qquad y = (W_0 + sBA)x,$$2-space by

$$\Delta W = sBA, \qquad W = W_0 + \Delta W, \qquad y = (W_0 + sBA)x,$$3

The stated implication is that, under a first-order Taylor approximation, the inner maximization over the full space is matched by a single low-rank perturbation, thereby aligning LoRA training with SAM’s full-space robust objective while avoiding interference across multiple low-rank matrices (Deng et al., 1 Aug 2025).

The analysis also reports a balancedness result. Let $$\Delta W = sBA, \qquad W = W_0 + \Delta W, \qquad y = (W_0 + sBA)x,$$4 and $$\Delta W = sBA, \qquad W = W_0 + \Delta W, \qquad y = (W_0 + sBA)x,$$5, and define the balancedness quantity as

$$\Delta W = sBA, \qquad W = W_0 + \Delta W, \qquad y = (W_0 + sBA)x,$$6

Under the Flat-LoRA update, the limiting flow for $$\Delta W = sBA, \qquad W = W_0 + \Delta W, \qquad y = (W_0 + sBA)x,$$7 satisfies

$$\Delta W = sBA, \qquad W = W_0 + \Delta W, \qquad y = (W_0 + sBA)x,$$8

with the conclusion that reducing $$\Delta W = sBA, \qquad W = W_0 + \Delta W, \qquad y = (W_0 + sBA)x,$$9 during training improves balancedness, which the paper relates to stability and scale-invariance (Deng et al., 1 Aug 2025).

4. Algorithmic forms: Flat-LoRA and EFlat-LoRA

The algorithm named Flat-LoRA uses a two-pass robust training procedure analogous to SAM, but with perturbation transfer. One iteration consists of: computing $B \in \mathbb{R}^{n \times r}$0 and $B \in \mathbb{R}^{n \times r}$1 at the current point $B \in \mathbb{R}^{n \times r}$2; building $B \in \mathbb{R}^{n \times r}$3 and $B \in \mathbb{R}^{n \times r}$4; transferring to $B \in \mathbb{R}^{n \times r}$5 and forming the perturbed point $B \in \mathbb{R}^{n \times r}$6; then performing a second forward/backward pass at $B \in \mathbb{R}^{n \times r}$7 and updating $B \in \mathbb{R}^{n \times r}$8 with an optimizer such as AdamW. The pass count is therefore two forward/backward passes per step, like SAM (Deng et al., 1 Aug 2025).

EFlat-LoRA is the efficient variant. It replaces the fresh second-pass perturbation with an exponential moving average of low-rank perturbations:

$B \in \mathbb{R}^{n \times r}$9

Each iteration evaluates gradients at the EMA-perturbed point

$A \in \mathbb{R}^{r \times m}$0

uses those gradients to estimate $A \in \mathbb{R}^{r \times m}$1 and compute the current transfer $A \in \mathbb{R}^{r \times m}$2, then returns to the original parameters and performs a single optimizer update using the gradients from the EMA-perturbed point. The EMA is updated afterward, and the next perturbed point is constructed for the next step. The pass count is one forward/backward pass, so the runtime is intended to be near standard LoRA (Deng et al., 1 Aug 2025).

The one-pass approximation is supported by a theorem comparing EMA sharpness

$A \in \mathbb{R}^{r \times m}$3

with SAM sharpness

$A \in \mathbb{R}^{r \times m}$4

under the assumptions that $A \in \mathbb{R}^{r \times m}$5 is $A \in \mathbb{R}^{r \times m}$6-smooth, gradients are bounded by $A \in \mathbb{R}^{r \times m}$7, gradient variance is bounded by $A \in \mathbb{R}^{r \times m}$8, local convexity holds during fine-tuning, and the perturbation radius decays as $A \in \mathbb{R}^{r \times m}$9. The reported bound is

$s>0$0

which is presented as justification that the EMA perturbation asymptotically approximates the SAM perturbation (Deng et al., 1 Aug 2025).

The practical design choices follow standard LoRA recipes wherever possible. The paper recommends standard LoRA settings for rank $s>0$1 and scaling $s>0$2, AdamW as optimizer, the usual LoRA regularization pipeline, a task-dependent perturbation radius $s>0$3, and a decaying schedule $s>0$4. For EFlat-LoRA, EMA momentum $s>0$5 is taken in $s>0$6, with values around $s>0$7 described as typical (Deng et al., 1 Aug 2025).

5. Computational profile and implementation characteristics

The parameter count of LoRA, Flat-LoRA, and EFlat-LoRA is the same order per layer:

$s>0$8

Thus the methods do not increase the number of trainable parameters relative to ordinary LoRA (Deng et al., 1 Aug 2025).

The memory overhead differs between the two variants. The two-pass Flat-LoRA scheme stores original $s>0$9, $A$0, and $A$1-gradients transiently, giving

$A$2

EFlat-LoRA maintains an EMA perturbation $A$3, giving

$A$4

which is described as comparable to AdamW’s moments already stored for LoRA parameters (Deng et al., 1 Aug 2025).

The time complexity follows directly from the number of passes:

$A$5

Measured wall-clock results on GPT-2 fine-tuning for E2E NLG are reported as follows. For GPT-2 Medium, LoRA uses 23.6 GB and 4.30 h, Flat-LoRA 24.0 GB and 9.10 h, and EFlat-LoRA 24.0 GB and 4.80 h. For GPT-2 Large, LoRA uses 23.2 GB and 8.45 h, Flat-LoRA 23.6 GB and 17.47 h, and EFlat-LoRA 23.2 GB and 10.00 h. The conclusion drawn in the paper is that Flat-LoRA has roughly $A$6 LoRA runtime, whereas EFlat-LoRA is approximately $A$7–$A$8 LoRA runtime with essentially identical memory usage (Deng et al., 1 Aug 2025).

The earlier Flat-LoRA paper emphasizes a related implementation goal from a different angle: it seeks flatness in the full parameter space while preserving training efficiency, avoids the significant computation and memory overheads of standard sharpness-aware minimization, and manages memory overhead using random seeds (Li et al., 2024). Taken together, these descriptions indicate that computational efficiency is not incidental but part of the method’s definition.

6. Empirical behavior, generalization, and scope

Across the reported experiments, Flat-LoRA and EFlat-LoRA are presented as methods that improve generalization across diverse modalities and task families. The original Flat-LoRA paper states that experiments on mathematical reasoning, coding abilities, dialogue generation, instruction following, and text-to-image generation demonstrate improvements in both in-domain and out-of-domain generalization (Li et al., 2024).

The more detailed experimental account reports results for few-shot NLU with RoBERTa-large (355M) on SST-2, SST-5, SNLI, MNLI, RTE, and TREC. The average scores are: LoRA 80.0, LoRA-SAM 81.3, oBAR 80.9, nBAR 81.2, Flat-LoRA 83.1, and EFlat-LoRA 82.3. The highlighted task-level numbers include Flat-LoRA at 95.1 ± 0.5 on SST-2, 86.4 ± 0.8 on SNLI, and 82.7 ± 1.0 on MNLI (Deng et al., 1 Aug 2025).

On GLUE fine-tuning with RoBERTa-large, the reported development averages are FT 88.9, LoRA 88.4, oBAR 88.7, nBAR 88.9, and EFlat-LoRA 89.4. The EFlat-LoRA task means are SST-2 96.3 ± 0.2, STS-B 92.7 ± 0.1, RTE 89.3 ± 0.6, QQP 91.6 ± 0.1, QNLI 94.8 ± 0.1, MRPC 91.5 ± 0.4, MNLI 90.7 ± 0.1, and CoLA 68.0 ± 1.2. The paper states that EFlat-LoRA outperforms standard LoRA by +1.0 average and matches or exceeds full fine-tuning on several tasks (Deng et al., 1 Aug 2025).

For GPT-2 on E2E NLG, EFlat-LoRA is reported to consistently match or outperform LoRA without increasing trainable parameters. With GPT-2 Medium, LoRA versus EFlat-LoRA changes from BLEU 69.2 to 69.7, NIST 8.72 to 8.77, MET 46.5 to 46.6, ROUGE-L 71.5 to 71.7, and CIDEr 2.51 to 2.53. With GPT-2 Large, the corresponding changes are BLEU 69.9 to 70.2, NIST 8.82 to 8.84, MET 46.8 to 46.8, ROUGE-L 71.8 to 71.8, and CIDEr 2.53 to 2.52 (Deng et al., 1 Aug 2025).

The method is also evaluated in vision-language settings. For few-shot CLIP adaptation on EuroSAT, OxfordPets, Flowers102, Caltech101, and DTD, EFlat-LoRA excels on EuroSAT in the 1-shot setting with 78.3, ties best on Pets with 92.8, and Flat-LoRA or EFlat-LoRA outperform Adapter and LoRA in most 4-shot and 16-shot settings. For Qwen-VL-Chat fine-tuning, the LoRA baseline is 90.1 on ScienceQA and 50.69 on VizWiz, while EFlat-LoRA with $A$9 reaches 91.6 on ScienceQA and with $B$0 reaches 51.7 on VizWiz (Deng et al., 1 Aug 2025).

Several broader conclusions are drawn in the paper. First, the experiments are used to verify that the generalization of LoRA is closely related to sharpness. Second, the perturbation radius $B$1 is a critical hyperparameter, and the sensitivity observed on Qwen-VL-Chat supports the recommendation to use a task-dependent perturbation scale together with the decaying schedule $B$2. Third, the method is intended to be drop-in with standard LoRA pipelines: reported practice is to reuse the baseline LoRA optimizer, learning rate, weight decay, dropout, and clipping settings (Deng et al., 1 Aug 2025).

A common misconception is that sharpness-aware fine-tuning for LoRA is adequately handled by perturbing both low-rank factors with SAM-style updates. The Flat-LoRA analysis argues that this is precisely where the misalignment occurs, because independent perturbations in $B$3 and $B$4 distort the inner maximization relative to the full parameter space. Another misconception is that flatness claims concern only local geometry inside the low-rank manifold; Flat-LoRA is explicitly designed to make the robustness criterion refer to the combined full-space weight $B$5 (Deng et al., 1 Aug 2025).

Within the reported evidence, Flat-LoRA therefore occupies a specific place in the PEFT landscape: it is a LoRA-compatible mechanism for seeking flat minima in the full parameter space, with a two-pass version that mirrors SAM more directly and an EMA-based one-pass version that aims to preserve most of LoRA’s efficiency (Li et al., 2024, Deng et al., 1 Aug 2025).