---
title: 'Flat-LoRA: Full-Parameter Fine-Tuning'
url: https://www.emergentmind.com/topics/flat-lora
type: topic
---

# Flat-LoRA: Full-Parameter Fine-Tuning

Flat-LoRA is a Parameter-Efficient Fine-Tuning (PEFT) method for large pre-trained models that seeks a low-rank adaptation located in a flat region of the **full parameter space**, rather than merely a flat region inside the constrained LoRA optimization space. In the original formulation, the motivating observation is that a solution that appears flat in the loss landscape of the LoRA space may still exhibit sharp directions in the full parameter space, which can compromise generalization; Flat-LoRA addresses this without adopting the standard sharpness-aware minimization procedure with its associated computation and memory overheads, instead using a Bayesian expectation loss objective and a refined random perturbation generation strategy [2409.14396]. Subsequent work gives a more explicit sharpness-aware formulation, shows how perturbations in the full parameter space can be transferred to the low-rank subspace, and introduces EFlat-LoRA as an efficient variant with near-LoRA optimization cost [2508.00522].

## 1. Problem setting and motivating intuition

Flat-LoRA arises from a specific limitation of standard LoRA. LoRA constrains training to a low-dimensional subspace of updates, so that only low-rank matrices are optimized while the pre-trained backbone remains frozen. This makes fine-tuning efficient in memory and computation, but it also means that the geometry seen during optimization is only a projection of the geometry of the full model. The central claim behind Flat-LoRA is that **flatness in the LoRA subspace does not guarantee flatness in the full parameter space** [2409.14396].

The later theoretical treatment makes the point more sharply in terms of sharpness-aware optimization. A long line of theory and empirical evidence links flatter minima to better generalization, and SAM improves generalization by optimizing a robust objective based on adversarial weight perturbations. In LoRA, however, directly applying SAM to the two low-rank factors can create a mismatch between the perturbation neighborhood in the low-rank parameterization and the intended neighborhood in the full weight matrix. The proposed Flat-LoRA framework is therefore motivated by the need to seek flat minima for LoRA **in a way aligned with the full parameter space**, rather than by perturbing LoRA factors independently [2508.00522].

This framing also clarifies what Flat-LoRA is not. It is not merely a regularized LoRA variant that prefers small updates; rather, it is a procedure for making low-rank adaptation responsive to full-space sharpness. This suggests that its contribution is geometric as much as parametric: the method changes the effective neighborhood in which robustness is optimized.

## 2. LoRA parameterization and the mismatch with naïve SAM

The formal setup starts from a frozen base weight matrix $W_0 \in \mathbb{R}^{n \times m}$. LoRA introduces a rank-$r$ update with $r \ll \min(n,m)$ through two trainable factors,
$$
\Delta W = sBA, \qquad W = W_0 + \Delta W, \qquad y = (W_0 + sBA)x,
$$
where $B \in \mathbb{R}^{n \times r}$ is the up-projection, $A \in \mathbb{R}^{r \times m}$ is the down-projection, and $s>0$ is the LoRA scaling hyperparameter. The reported initialization is standard: $A$ is Kaiming-initialized, $B$ is zero-initialized, $W_0$ is frozen, and only $A$ and $B$ are trainable [2508.00522].

SAM optimizes a robust objective of the form
$$
\min_w \max_{\|\epsilon\| \le \rho} L(w+\epsilon),
$$
with the first-order perturbation approximation
$$
\hat{\epsilon}(w) = \rho \frac{\nabla_w L(w)}{\|\nabla_w L(w)\|_2}.
$$
Its sharpness proxy is the perturbed loss increase $L(w+\hat{\epsilon}) - L(w)$, and the standard implementation requires two forward/backward passes [2508.00522].

The difficulty appears when this is applied directly to the LoRA factors. A naïve LoRA-only robust objective can be written as
$$
\min_{A,B} \max_{\|E^A\|_F \le \rho,\ \|E^B\|_F \le \rho}
L\!\left(W_0 + s(B+E^B)(A+E^A)\right).
$$
The crucial issue is the cross-term $sE^B E^A$. According to the theoretical analysis, this term introduces interference between the two low-rank subspaces and makes the inner maximization inconsistent with the full-parameter inner maximization around $W=W_0+sBA$ [2508.00522].

That interference is the main reason Flat-LoRA does not simply add SAM on top of LoRA. The method instead attempts to preserve the semantics of full-space adversarial perturbation while remaining inside the low-rank parameterization.

## 3. Perturbation transfer from the full parameter space

The core theoretical contribution is a perturbation-transfer construction. Rather than perturbing both LoRA factors independently, the method starts from the full-space robust objective
$$
\min_{A,B} \max_{\|E^W\|_F \le \rho} L(W_0 + sBA + E^W),
$$
and then transfers the full-space perturbation $E^W$ to a perturbation on only one low-rank factor, typically $B$, while fixing $A$ so that $E^A=0$ [2508.00522].

Under a first-order approximation, the transfer is implemented with the Moore–Penrose pseudo-inverse of $A$:
$$
E^B \approx \frac{1}{s} E^W A^+.
$$
The stated intuition is that
$$
s(B+E^B)A = sBA + sE^B A \approx sBA + E^W,
$$
which yields a first-order loss match and eliminates the problematic $E^B E^A$ cross-term by construction [2508.00522].

Because LoRA training does not explicitly store or directly compute $\nabla_W L(W)$, the full-space gradient is estimated from LoRA gradients in two first-order consistent ways:
$$
\nabla_W L(W) \approx \frac{1}{s}\nabla_B L(W_0+sBA)(A^T)^+,
$$
and
$$
\nabla_W L(W) \approx \frac{1}{s}(B^T)^+\nabla_A L(W_0+sBA).
$$
For stability, the two are averaged:
$$
\overline{\nabla}_W L(W) =
\frac{1}{2s}\left[\nabla_B L(W_0+sBA)(A^T)^+ + (B^T)^+\nabla_A L(W_0+sBA)\right].
$$
A SAM-style full-parameter perturbation is then defined in Frobenius norm as
$$
\bar{E}^W = \rho \frac{\overline{\nabla}_W L(W)}{\|\overline{\nabla}_W L(W)\|_F},
$$
and transferred to the LoRA $B$-space by
$$
E^B \approx \frac{1}{s}\bar{E}^W A^+.
$$
The stated implication is that, under a first-order Taylor approximation, the inner maximization over the full space is matched by a single low-rank perturbation, thereby aligning LoRA training with SAM’s full-space robust objective while avoiding interference across multiple low-rank matrices [2508.00522].

The analysis also reports a balancedness result. Let $x_t=\mathrm{Vector}(B_t)$ and $y_t=\mathrm{Vector}(A_t)$, and define the balancedness quantity as
$$
\mathcal{B}_t = \frac{1}{2}\left(\|x_t\|^2 - \|y_t\|^2\right).
$$
Under the Flat-LoRA update, the limiting flow for $\eta \to 0$ satisfies
$$
\left| \frac{1}{2}\frac{d}{dt}\left(\|x_t\|^2-\|y_t\|^2\right) \right|
\le \frac{\rho}{s}\left(\frac{\|g_{\tilde{x}_t}\|}{\|y_t\|}\right),
$$
with the conclusion that reducing $\rho$ during training improves balancedness, which the paper relates to stability and scale-invariance [2508.00522].

## 4. Algorithmic forms: Flat-LoRA and EFlat-LoRA

The algorithm named Flat-LoRA uses a two-pass robust training procedure analogous to SAM, but with perturbation transfer. One iteration consists of: computing $\nabla_A L$ and $\nabla_B L$ at the current point $W=W_0+sBA$; building $\overline{\nabla}_W$ and $\bar{E}^W$; transferring to $E^B \leftarrow (1/s)\bar{E}^W A^+$ and forming the perturbed point $\tilde{W}=W_0+s(B+E^B)A$; then performing a second forward/backward pass at $\tilde{W}$ and updating $A,B$ with an optimizer such as AdamW. The pass count is therefore two forward/backward passes per step, like SAM [2508.00522].

EFlat-LoRA is the efficient variant. It replaces the fresh second-pass perturbation with an exponential moving average of low-rank perturbations:
$$
\hat{E}^B_t = (1-\beta)\hat{E}^B_{t-1} + \beta E^B_t.
$$
Each iteration evaluates gradients at the EMA-perturbed point
$$
\tilde{W}_{t-1} = W_0 + s(B_{t-1}+\hat{E}^B_{t-1})A_{t-1},
$$
uses those gradients to estimate $\overline{\nabla}_W$ and compute the current transfer $E^B_t=(1/s)\bar{E}^W A_{t-1}^+$, then returns to the original parameters and performs a single optimizer update using the gradients from the EMA-perturbed point. The EMA is updated afterward, and the next perturbed point is constructed for the next step. The pass count is one forward/backward pass, so the runtime is intended to be near standard LoRA [2508.00522].

The one-pass approximation is supported by a theorem comparing EMA sharpness
$$
S^{EMA} = L(w_t + \hat{\epsilon}_{t-1}) - L(w_t)
$$
with SAM sharpness
$$
S^{SAM} = L(w_t + \tilde{\epsilon}_t) - L(w_t),
$$
under the assumptions that $L$ is $\tau$-smooth, gradients are bounded by $G$, gradient variance is bounded by $\sigma^2$, local convexity holds during fine-tuning, and the perturbation radius decays as $\rho_t=\rho_0/\sqrt{t}$. The reported bound is
$$
|S^{EMA} - S^{SAM}|
\le
\left(\frac{\tau \rho_0}{\sqrt{t-1}} + G + \sigma^2\right)
\left(\frac{\rho_0}{\sqrt{t}} + \rho_0(1-\beta)^{t-1} + \rho_0\right),
$$
which is presented as justification that the EMA perturbation asymptotically approximates the SAM perturbation [2508.00522].

The practical design choices follow standard LoRA recipes wherever possible. The paper recommends standard LoRA settings for rank $r$ and scaling $s$, AdamW as optimizer, the usual LoRA regularization pipeline, a task-dependent perturbation radius $\rho$, and a decaying schedule $\rho_t=\rho_0/\sqrt{t}$. For EFlat-LoRA, EMA momentum $\beta$ is taken in $(0,1)$, with values around $0.9$ described as typical [2508.00522].

## 5. Computational profile and implementation characteristics

The parameter count of LoRA, Flat-LoRA, and EFlat-LoRA is the same order per layer:
$$
P_{\text{LoRA}} = P_{\text{Flat-LoRA}} = P_{\text{EFlat-LoRA}} = O(nr+rm) \ll O(nm).
$$
Thus the methods do not increase the number of trainable parameters relative to ordinary LoRA [2508.00522].

The memory overhead differs between the two variants. The two-pass Flat-LoRA scheme stores original $B$, $A$, and $A$-gradients transiently, giving
$$
M_{\text{Flat-LoRA}} = M_{\text{LoRA}} + O(1.5 \times (nr+rm)).
$$
EFlat-LoRA maintains an EMA perturbation $\hat{E}^B$, giving
$$
M_{\text{EFlat-LoRA}} = M_{\text{LoRA}} + O(2 \times (nr+rm)),
$$
which is described as comparable to AdamW’s moments already stored for LoRA parameters [2508.00522].

The time complexity follows directly from the number of passes:
$$
T_{\text{Flat-LoRA}} \approx 2T_{\text{LoRA}}, \qquad
T_{\text{EFlat-LoRA}} \approx T_{\text{LoRA}}.
$$
Measured wall-clock results on GPT-2 fine-tuning for E2E NLG are reported as follows. For GPT-2 Medium, LoRA uses 23.6 GB and 4.30 h, Flat-LoRA 24.0 GB and 9.10 h, and EFlat-LoRA 24.0 GB and 4.80 h. For GPT-2 Large, LoRA uses 23.2 GB and 8.45 h, Flat-LoRA 23.6 GB and 17.47 h, and EFlat-LoRA 23.2 GB and 10.00 h. The conclusion drawn in the paper is that Flat-LoRA has roughly $2\times$ LoRA runtime, whereas EFlat-LoRA is approximately $1.1$–$1.2\times$ LoRA runtime with essentially identical memory usage [2508.00522].

The earlier Flat-LoRA paper emphasizes a related implementation goal from a different angle: it seeks flatness in the full parameter space while preserving training efficiency, avoids the significant computation and memory overheads of standard sharpness-aware minimization, and manages memory overhead using random seeds [2409.14396]. Taken together, these descriptions indicate that computational efficiency is not incidental but part of the method’s definition.

## 6. Empirical behavior, generalization, and scope

Across the reported experiments, Flat-LoRA and EFlat-LoRA are presented as methods that improve generalization across diverse modalities and task families. The original Flat-LoRA paper states that experiments on mathematical reasoning, coding abilities, dialogue generation, instruction following, and text-to-image generation demonstrate improvements in both in-domain and out-of-domain generalization [2409.14396].

The more detailed experimental account reports results for few-shot NLU with RoBERTa-large (355M) on SST-2, SST-5, SNLI, MNLI, RTE, and TREC. The average scores are: LoRA 80.0, LoRA-SAM 81.3, oBAR 80.9, nBAR 81.2, Flat-LoRA 83.1, and EFlat-LoRA 82.3. The highlighted task-level numbers include Flat-LoRA at 95.1 ± 0.5 on SST-2, 86.4 ± 0.8 on SNLI, and 82.7 ± 1.0 on MNLI [2508.00522].

On GLUE fine-tuning with RoBERTa-large, the reported development averages are FT 88.9, LoRA 88.4, oBAR 88.7, nBAR 88.9, and EFlat-LoRA 89.4. The EFlat-LoRA task means are SST-2 96.3 ± 0.2, STS-B 92.7 ± 0.1, RTE 89.3 ± 0.6, QQP 91.6 ± 0.1, QNLI 94.8 ± 0.1, MRPC 91.5 ± 0.4, MNLI 90.7 ± 0.1, and CoLA 68.0 ± 1.2. The paper states that EFlat-LoRA outperforms standard LoRA by +1.0 average and matches or exceeds full fine-tuning on several tasks [2508.00522].

For GPT-2 on E2E NLG, EFlat-LoRA is reported to consistently match or outperform LoRA without increasing trainable parameters. With GPT-2 Medium, LoRA versus EFlat-LoRA changes from BLEU 69.2 to 69.7, NIST 8.72 to 8.77, MET 46.5 to 46.6, ROUGE-L 71.5 to 71.7, and CIDEr 2.51 to 2.53. With GPT-2 Large, the corresponding changes are BLEU 69.9 to 70.2, NIST 8.82 to 8.84, MET 46.8 to 46.8, ROUGE-L 71.8 to 71.8, and CIDEr 2.53 to 2.52 [2508.00522].

The method is also evaluated in vision-language settings. For few-shot CLIP adaptation on EuroSAT, OxfordPets, Flowers102, Caltech101, and DTD, EFlat-LoRA excels on EuroSAT in the 1-shot setting with 78.3, ties best on Pets with 92.8, and Flat-LoRA or EFlat-LoRA outperform Adapter and LoRA in most 4-shot and 16-shot settings. For Qwen-VL-Chat fine-tuning, the LoRA baseline is 90.1 on ScienceQA and 50.69 on VizWiz, while EFlat-LoRA with $\rho=0.2$ reaches 91.6 on ScienceQA and with $\rho=0.05$ reaches 51.7 on VizWiz [2508.00522].

Several broader conclusions are drawn in the paper. First, the experiments are used to verify that the generalization of LoRA is closely related to sharpness. Second, the perturbation radius $\rho$ is a critical hyperparameter, and the sensitivity observed on Qwen-VL-Chat supports the recommendation to use a task-dependent perturbation scale together with the decaying schedule $\rho_t=\rho_0/\sqrt{t}$. Third, the method is intended to be drop-in with standard LoRA pipelines: reported practice is to reuse the baseline LoRA optimizer, learning rate, weight decay, dropout, and clipping settings [2508.00522].

A common misconception is that sharpness-aware fine-tuning for LoRA is adequately handled by perturbing both low-rank factors with SAM-style updates. The Flat-LoRA analysis argues that this is precisely where the misalignment occurs, because independent perturbations in $A$ and $B$ distort the inner maximization relative to the full parameter space. Another misconception is that flatness claims concern only local geometry inside the low-rank manifold; Flat-LoRA is explicitly designed to make the robustness criterion refer to the combined full-space weight $W=W_0+sBA$ [2508.00522].

Within the reported evidence, Flat-LoRA therefore occupies a specific place in the PEFT landscape: it is a LoRA-compatible mechanism for seeking flat minima in the full parameter space, with a two-pass version that mirrors SAM more directly and an EMA-based one-pass version that aims to preserve most of LoRA’s efficiency [2409.14396; 2508.00522].

Source: https://www.emergentmind.com/topics/flat-lora