AS-LoRA: Adaptive Federated Fine-Tuning

• The paper introduces an adaptive federated fine-tuning method that selects the active LoRA factor based on a curvature-aware second-order score.
• It mitigates aggregation mismatch and DP-noise amplification by updating either A or B per layer and round, leading to improved convergence and flatter solutions.
• Empirical results across language and vision tasks demonstrate that AS-LoRA outperforms standard LoRA schedules with minimal communication and computational overhead.

Searching arXiv for papers on AS-LoRA and closely related LoRA variants to ground the article in the cited literature. arxiv_search query: "AS-LoRA Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning"

AS-LoRA is a federated fine-tuning method for low-rank adaptation under differential privacy. It studies the setting in which a large pretrained model is adapted across multiple clients without sharing raw data, while local optimization is protected with DP-SGD. Its defining mechanism is the adaptive choice, for each LoRA-applied layer and for each communication round, of which LoRA component is active: $A$ or $B$. That choice is driven by a curvature-aware score derived from a second-order approximation of the loss, with the stated goals of reducing aggregation mismatch, mitigating DP-noise amplification, improving convergence, and favoring flatter solutions (Kim et al., 7 May 2026).

1. Definition, scope, and nomenclature

AS-LoRA is introduced as Adaptive Selection LoRA. The method adapts along three axes: layer-wise freedom, in which each layer can choose a different active LoRA component; round-wise adaptivity, in which the choice can change over communication rounds; and curvature-aware scoring, in which the active component is selected by a score that estimates one-step loss decrease while penalizing sharp directions (Kim et al., 7 May 2026).

The immediate target problem is not generic parameter-efficient fine-tuning, but privacy-preserving federated fine-tuning. The base model is frozen except for LoRA parameters, client updates are computed locally, and the server performs round-based aggregation. In this setting, AS-LoRA treats the asymmetry between the two low-rank factors as a scheduling variable rather than as a fixed design choice.

A naming distinction is important. A separate method, "ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layers," addresses cross-layer parameter sharing by globally sharing one LoRA factor and progressively merging the other across transformer layers (Hu et al., 2024). Despite the similar name, that method is organized around cross-layer sharing within a single training run, whereas AS-LoRA is organized around layer-wise and round-wise factor selection in federated learning under DP.

2. Problem setting and motivation

AS-LoRA is motivated by a structural mismatch between standard LoRA and standard federated aggregation. For a pretrained weight matrix, the paper writes

$W = W_0 + BA,$

where $B \in \mathbb{R}^{d_{\text{out}} \times r}$, $A \in \mathbb{R}^{r \times d_{\text{in}}}$, and $r \ll \min(d_{\text{out}}, d_{\text{in}})$. In the experiments, LoRA rank is $r=8$, scaling factor $\alpha=8$, LoRA is applied to query and value projections, dropout is $0.05$, and no bias terms are used (Kim et al., 7 May 2026).

Under federated averaging, if client $k$ has factors $B$0, the server forms

$B$1

But the effective update is $B$2, and in general

$B$3

This is the aggregation error emphasized by the paper: averaging factors and then multiplying is not the same as averaging the low-rank products.

Differential privacy worsens the problem. With DP-SGD, per-sample gradients are clipped and perturbed,

$B$4

and

$B$5

The experiments use clipping norm $B$6, $B$7, and privacy budgets $B$8 in the main results. The paper writes the induced effective weight update as

$B$9

The last term, $W = W_0 + BA,$0, is identified as a multiplicative cross-noise term whose norm grows quadratically in the noise scale $W = W_0 + BA,$1, in contrast to ordinary full-model noise, which grows linearly (Kim et al., 7 May 2026).

This diagnosis is used to critique earlier federated LoRA schedules. FedLoRA trains both $W = W_0 + BA,$2 and $W = W_0 + BA,$3 and averages them directly, making it most exposed to aggregation mismatch. FFA-LoRA freezes $W = W_0 + BA,$4 and trains only $W = W_0 + BA,$5, reducing mismatch but sacrificing expressiveness. RoLoRA alternates globally between $W = W_0 + BA,$6-only and $W = W_0 + BA,$7-only rounds, which is more expressive than freezing one factor but still enforces the same mode for every layer at a given round (Kim et al., 7 May 2026).

3. Mathematical formulation and adaptive selection rule

For the $W = W_0 + BA,$8-th LoRA layer at round $W = W_0 + BA,$9, AS-LoRA writes

$B \in \mathbb{R}^{d_{\text{out}} \times r}$0

The method introduces a mode vector

$B \in \mathbb{R}^{d_{\text{out}} \times r}$1

with

• $B \in \mathbb{R}^{d_{\text{out}} \times r}$2: optimize $B \in \mathbb{R}^{d_{\text{out}} \times r}$3,
• $B \in \mathbb{R}^{d_{\text{out}} \times r}$4: optimize $B \in \mathbb{R}^{d_{\text{out}} \times r}$5.

Thus, for each layer and round, only one factor is active, updated locally, and aggregated globally. If $B \in \mathbb{R}^{d_{\text{out}} \times r}$6 is active for layer $B \in \mathbb{R}^{d_{\text{out}} \times r}$7,

$B \in \mathbb{R}^{d_{\text{out}} \times r}$8

and if $B \in \mathbb{R}^{d_{\text{out}} \times r}$9 is active,

$A \in \mathbb{R}^{r \times d_{\text{in}}}$0

This avoids averaging both factors simultaneously in the same layer-round pair (Kim et al., 7 May 2026).

The active component is chosen by a second-order score. For a candidate block $A \in \mathbb{R}^{r \times d_{\text{in}}}$1 at layer $A \in \mathbb{R}^{r \times d_{\text{in}}}$2, the paper uses the Taylor approximation

$A \in \mathbb{R}^{r \times d_{\text{in}}}$3

which yields

$A \in \mathbb{R}^{r \times d_{\text{in}}}$4

The first term favors blocks with larger gradient energy; the second penalizes high curvature. In Rayleigh-quotient form,

$A \in \mathbb{R}^{r \times d_{\text{in}}}$5

so

$A \in \mathbb{R}^{r \times d_{\text{in}}}$6

This expresses the paper’s flatness interpretation directly: for comparable gradient norms, a block with smaller directional curvature receives the higher score (Kim et al., 7 May 2026).

Two practical score approximations are given. The first uses Hessian-vector products:

$A \in \mathbb{R}^{r \times d_{\text{in}}}$7

with the corresponding scores

$A \in \mathbb{R}^{r \times d_{\text{in}}}$8

$A \in \mathbb{R}^{r \times d_{\text{in}}}$9

The second uses finite differences. Let

$r \ll \min(d_{\text{out}}, d_{\text{in}})$0

and estimate directional curvature by

$r \ll \min(d_{\text{out}}, d_{\text{in}})$1

Then

$r \ll \min(d_{\text{out}}, d_{\text{in}})$2

The main experiments use FD-based scoring, not HVP, for practicality (Kim et al., 7 May 2026).

A further practical device is Gaussian random projection:

$r \ll \min(d_{\text{out}}, d_{\text{in}})$3

Its stated role is not standard dimensionality reduction; rather, it preserves dimensionality while redistributing severe channel-wise outliers, especially in LoRA-$r \ll \min(d_{\text{out}}, d_{\text{in}})$4 gradients under small-batch DP-SGD (Kim et al., 7 May 2026).

4. Federated algorithm, privacy accounting, and complexity

At each communication round, the server maintains global LoRA parameters and score statistics, computes the current mode vector, and broadcasts the model together with mode assignments. During warm-up, the paper uses simple alternating activation across rounds,

$r \ll \min(d_{\text{out}}, d_{\text{in}})$5

After warm-up, the server computes selection probabilities

$r \ll \min(d_{\text{out}}, d_{\text{in}})$6

with temperature schedule

$r \ll \min(d_{\text{out}}, d_{\text{in}})$7

The deployed algorithm then samples

$r \ll \min(d_{\text{out}}, d_{\text{in}})$8

The reported hyperparameters are warm-up ratio $r \ll \min(d_{\text{out}}, d_{\text{in}})$9, $r=8$0, $r=8$1, and $r=8$2 (Kim et al., 7 May 2026).

Selection is layer-wise and shared across clients, not per-client. The paper explicitly compares three granularities: per-client selection, one global mode for all layers and clients, and the AS-LoRA design in which one mode is chosen per layer and shared across clients. The rationale is that per-client modes would make the server aggregate inconsistent objects, reintroducing instability.

The paper states that AS-LoRA incurs no extra privacy cost beyond standard DP-SGD. Theorem 1 gives the same $r=8$3-DP bound as standard subsampled Gaussian DP-SGD,

$r=8$4

and attributes this to post-processing invariance: mode selection is computed from already private outputs (Kim et al., 7 May 2026).

Communication overhead is described as negligible. If

$r=8$5

then FedLoRA communicates $r=8$6, FFA-LoRA communicates $r=8$7, RoLoRA communicates either $r=8$8 or $r=8$9, and AS-LoRA communicates only one active component per layer, i.e. the same order as alternating methods. Its extra overhead is $\alpha=8$0 scalar scores per client per round on the uplink and $\alpha=8$1 bits for mode indicators on the downlink. For $\alpha=8$2 layers, this is $\alpha=8$3 FP32 values, or $\alpha=8$4 bytes per client per round, stated to be less than $\alpha=8$5 of active LoRA traffic (Kim et al., 7 May 2026).

The main computational trade-off is curvature scoring. The appendix reports:

• baseline training:

$\alpha=8$6

for $\alpha=8$7;

• full central-difference curvature scoring with $\alpha=8$8 targets:

$\alpha=8$9

i.e. a $0.05$0 overhead over baseline.

To reduce cost, the paper studies one-sided FD and occasional curvature computation. With one-sided FD and periodic scoring every $0.05$1 rounds,

$0.05$2

which is reported as only $0.05$3 overhead and a $0.05$4 reduction from the original curvature cost (Kim et al., 7 May 2026).

5. Theoretical claims and empirical results

The theory centers on reconstruction risk and adaptive gain. The paper defines row-space misalignment for layer $0.05$5 as

$0.05$6

where

$0.05$7

is the orthogonal projector onto $0.05$8. The layer-averaged reconstruction risk is

$0.05$9

Theorem 2 states that if $k$0 is frozen, FFA-LoRA retains an irreducible floor:

$k$1

The paper also argues that any RoLoRA-type layer-tied alternation inherits

$k$2

under adversarial layer configurations. By contrast, Corollary 1 states that if AS-LoRA activates $k$3 on $k$4 rounds per layer, then for any $k$5 there exists

$k$6

such that

$k$7

and

$k$8

This is the basis for the claim that AS-LoRA eliminates the reconstruction-error floor of layer-tied schedules (Kim et al., 7 May 2026).

The empirical study uses RoBERTa-large on GLUE and SQuAD, ViT-large on CIFAR-100 and Tiny-ImageNet, $k$9 clients for language tasks, $B$00 for vision, $B$01 communication rounds, $B$02 local steps per round, batch size $B$03, and Dirichlet partitioning with $B$04 for most tasks (Kim et al., 7 May 2026).

The headline results are as follows.

Setting	Best non-AS-LoRA baseline	AS-LoRA
GLUE average, $B$05	75.27 (RoLoRA)	82.81
GLUE average, $B$06	71.89 (FFA-LoRA)	77.63
MNLI-mm, $B$07	65.04 (FFA-LoRA)	77.54
CIFAR-100, $B$08	87.75 (RoLoRA)	89.55

On GLUE at $B$09, the averages are 73.10 for FedLoRA, 74.24 for FFA-LoRA, 75.27 for RoLoRA, and 82.81 for AS-LoRA, a gain of $B$10 percentage points over the best baseline. On the same privacy budget, AS-LoRA reports 81.06 on QNLI, 79.15 on MNLI-m, and 80.03 on MNLI-mm. At $B$11, the averages are 61.87, 71.89, 65.37, and 77.63, respectively; on MNLI-mm, AS-LoRA reaches 77.54 against 65.04 for FFA-LoRA, a gain of $B$12 percentage points (Kim et al., 7 May 2026).

On question answering and vision, AS-LoRA remains best or near-best. At $B$13, SQuAD v1.1 yields 73.98 EM / 83.99 F1 for AS-LoRA against 72.30 / 82.83 for FFA-LoRA. SQuAD v2.0 yields 60.59 / 64.23 for AS-LoRA against 55.90 / 59.92 for FFA-LoRA. CIFAR-100 yields 89.55 for AS-LoRA against 87.75 for RoLoRA, and Tiny-ImageNet yields 86.66 against 85.50 for FFA-LoRA. At $B$14, AS-LoRA reports 72.78/83.23 on SQuAD v1.1, 58.09/61.64 on SQuAD v2.0, 85.62 on CIFAR-100, and 83.71 on Tiny-ImageNet (Kim et al., 7 May 2026).

Ablations refine the picture. For selection granularity on MNLI-m/mm at $B$15, per-client selection gives 72.73/73.73, a single global mode gives 75.74/76.98, and AS-LoRA gives 79.15/80.03. At $B$16, per-client selection collapses to 34.60/34.53, which the paper reads as evidence that per-client freedom destroys aggregation consistency. For curvature approximation, FD gives 79.15/80.03 while HVP gives 80.29/81.29, indicating a slight quality advantage for HVP but with lower practicality. Periodic scoring every $B$17 rounds gives 78.83/79.76, showing that full curvature evaluation is not necessary every round. Random projection improves QQP under $B$18, especially at smaller batch sizes: for batch 32, 63.19 with RP versus 61.16 without; for batch 64, 73.99 versus 70.90; and for batch 128, 79.94 versus 79.15 (Kim et al., 7 May 2026).

The paper also compares AS-LoRA to FedSVD, an SVD-based server-side orthogonalization method. On MNLI at $B$19, FedSVD(10) reports 78.16 / 79.43, FedSVD(5) 77.06 / 79.07, FedSVD(1) 78.49 / 79.75, and AS-LoRA 79.15 / 80.03. Measured aggregation times on MNLI are 1.00 s for FedSVD(10), 1.80 s for FedSVD(5), 5.44 s for FedSVD(1), and 0.03 s for AS-LoRA, giving the reported $B$20 to $B$21 aggregation-cost advantage (Kim et al., 7 May 2026).

6. Interpretation, limitations, and relation to adjacent LoRA lines

AS-LoRA is best understood as an adaptive block-selection strategy for federated LoRA under privacy constraints. It does not alter the basic low-rank form $B$22, and it does not introduce a new server-side factor reconstruction such as SVD. Its novelty is the decision rule governing which factor is updated, where, and when, under the joint pressures of non-IID client drift and DP noise (Kim et al., 7 May 2026).

The paper’s flatness claim is supported both theoretically and empirically. Theorem 4 states that the expected top eigenvalue of the selected-block Hessian under AS-LoRA is no larger than under static schedules, with strict inequality where the two blocks’ spectra differ. Empirically, on MNLI the reported SAM-style perturbation sharpness is 0.03608 for FFA-LoRA, 0.01972 for RoLoRA, and 0.00992 for AS-LoRA, while the corresponding $B$23Acc values are 0.981, 0.321, and 0.247 (Kim et al., 7 May 2026). This suggests that the curvature penalty is functioning not only as a scheduling heuristic but also as an implicit regularizer.

The method also has clear limits. The paper explicitly notes that AS-LoRA makes independent layer-wise greedy decisions and does not model dependencies across layers, even though the appendix’s mode visualizations suggest cross-layer coordination may exist. Full curvature scoring can be expensive without scheduling tricks. Parts of the theory analyze an idealized argmax rule, whereas the deployed algorithm uses stochastic softmax sampling with temperature annealing. Random projection, EMA, and score-aggregation design are not fully incorporated into the formal analysis. The paper also remarks that weighted averaging or majority voting for score aggregation may outperform the default uniform averaging, but that is not the main reported method (Kim et al., 7 May 2026).

In the broader LoRA landscape, AS-LoRA occupies a distinct design point. It is unlike ASLoRA (Hu et al., 2024), whose central mechanism is cross-layer sharing of low-rank factors within a model; unlike Block-Diagonal LoRA (Wang et al., 27 Oct 2025), whose central concern is eliminating serving-time communication overhead under tensor parallelism; and unlike general empirical work on LoRA as modular knowledge memory (Back et al., 1 Mar 2026), whose focus is capacity, routing, and merge interference rather than federated DP aggregation. A plausible implication is that AS-LoRA belongs to a specialized sublineage of LoRA research in which the asymmetry of the two factors is operationalized as a control variable, rather than treated as a fixed parameterization detail.