Adaptive Rank Fine-Tuning with LR-LoRA

• The paper introduces LR-LoRA, a method that shifts from fixed global rank to learnable, layer-specific adaptation capacity in transformers.
• It employs a differentiable transfer function with sinc-basis parameterization to adjust low-rank updates, effectively capturing per-layer spectral complexity.
• Empirical results show that LR-LoRA improves performance over standard LoRA with negligible parameter and runtime overhead, demonstrating practical efficiency.

This paper, Parameter-Efficient Fine-Tuning with Learnable Rank, is highly relevant to the idea of Locally Adaptive Ranking (LAR), even though it does not use that exact term. Its central claim is that the rank constraint in standard LoRA should not be treated as a fixed, globally chosen hyperparameter. Instead, the model should be allowed to learn the effective adaptation dimensionality separately across layers and modules during fine-tuning. In that sense, the paper is best understood as proposing a layer-local, module-specific, continuously learned form of rank allocation.

Standard LoRA adapts a frozen pretrained weight matrix $\mathbf{W}\in\mathbb{R}^{m\times n}$ by adding a low-rank update: $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$ where $\mathbf{A}\in\mathbb{R}^{r\times n}$, $\mathbf{B}\in\mathbb{R}^{m\times r}$, and $r \ll \min(m,n)$. This yields the familiar parameter count $r(m+n)$, but it also imposes a strong inductive bias: $\mathrm{rank}(\mathbf{B}\mathbf{A}) \le r.$ The paper argues that this fixed-rank inductive bias is likely suboptimal because transformer layers differ substantially in function, depth, representation geometry, and intrinsic dimensionality. A single global $r$, usually applied uniformly across all adapted layers, implicitly assumes that all layers require the same adaptation capacity. The authors explicitly challenge this assumption and frame their contribution as a shift from “externally prescribing” adaptation dimensionality to learning it during optimization.

That is the key connection to LAR. In this paper, “local” means per adapted layer / per adapter instance, and more specifically per adapted projection matrix, because each adapter gets its own learned transfer function. It is not a single global rank, and it is not only per block unless multiple projections are tied manually in an ablation. The learned notion of rank is therefore module-local: attention and MLP projections in different depths can realize different effective ranks.

The proposed method, Learnable Rank LoRA (LR-LoRA), modifies LoRA by applying an elementwise learnable nonlinearity $\phi$ to the low-rank product: $\Delta \mathbf{W} = \phi(\mathbf{B}\mathbf{A}),$ so that the adapted weight becomes

$$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$0

The core insight is that while $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$1 is rank-constrained, $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$2 need not be. The paper’s claim is broad: any differentiable elementwise nonlinearity applied to $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$3 removes the strict rank-$$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$4 constraint. LR-LoRA instantiates this idea using a sinc-basis parameterization: $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$5 where $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$6 are uniformly spaced grid points on $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$7, and both amplitudes $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$8 and bandwidths $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$9 are trainable. The basis is motivated via sampling theory, but the authors repeatedly stress that their scientific point is basis-agnostic: the important thing is not sinc per se, but that a learnable differentiable transfer function lets the optimizer discover the layer’s needed adaptation complexity.

There is no explicit discrete rank-selection vector, no hard mask over singular directions, no thresholding rule, and no sparsity penalty that directly zeros components to produce a final integer rank. This is important for interpreting the method relative to adaptive-rank PEFT. LR-LoRA does not define something like a learnable vector $\mathbf{A}\in\mathbb{R}^{r\times n}$0 or soft gates over latent rank components, nor does it solve a constrained rank-budget allocation problem as in AdaLoRA-style methods. Instead, rank becomes a realized property of the update matrix $\mathbf{A}\in\mathbb{R}^{r\times n}$1, measured empirically through its spectrum. The paper uses stable rank rather than exact rank: $\mathbf{A}\in\mathbb{R}^{r\times n}$2 with $\mathbf{A}\in\mathbb{R}^{r\times n}$3 the Frobenius norm and $\mathbf{A}\in\mathbb{R}^{r\times n}$4 the spectral norm. The authors explicitly avoid exact rank because it is numerically threshold-dependent. Thus, “learnable rank” in LR-LoRA really means learnable effective spectral complexity, proxied by stable rank, not a directly optimized integer rank.

Methodologically, the model remains fully differentiable and trainable end-to-end because $\mathbf{A}\in\mathbb{R}^{r\times n}$5 is differentiable with respect to its parameters and with respect to its input $\mathbf{A}\in\mathbb{R}^{r\times n}$6. Each adapter has its own $\mathbf{A}\in\mathbb{R}^{r\times n}$7, so every adapted module learns separate transfer-function parameters. The initialization is chosen carefully: amplitudes are initialized to zero,

$\mathbf{A}\in\mathbb{R}^{r\times n}$8

so that initially

$\mathbf{A}\in\mathbb{R}^{r\times n}$9

and the model starts from the frozen backbone with no effective adapter perturbation. Bandwidths are parameterized via

$\mathbf{B}\in\mathbb{R}^{m\times r}$0

with $\mathbf{B}\in\mathbb{R}^{m\times r}$1 initialized so that $\mathbf{B}\in\mathbb{R}^{m\times r}$2 at the start; the default is $\mathbf{B}\in\mathbb{R}^{m\times r}$3. This ensures $\mathbf{B}\in\mathbb{R}^{m\times r}$4 and avoids degeneracy. The paper even explains why zero-initialized $\mathbf{B}\in\mathbb{R}^{m\times r}$5 does not kill learning: since

$\mathbf{B}\in\mathbb{R}^{m\times r}$6

the amplitudes receive nonzero gradients immediately, and once $\mathbf{B}\in\mathbb{R}^{m\times r}$7 moves away from zero, gradients propagate through $\mathbf{B}\in\mathbb{R}^{m\times r}$8 into $\mathbf{B}\in\mathbb{R}^{m\times r}$9 and $r \ll \min(m,n)$0.

The paper does not introduce any additional regularizer on rank, sparsity, or budget. There is no explicit penalty such as nuclear norm, $r \ll \min(m,n)$1 sparsity over rank gates, or KL term on capacity. The training objective is therefore just the ordinary downstream task loss optimized over the adapter factors and transfer-function parameters, with AdamW. The paper does not write the supervised task objective as an explicit formula, but it states that optimization schedules, preprocessing, and evaluation scripts are inherited from the RandLoRA codebase and kept identical across methods. So from the paper alone, the objective can only be summarized as the standard fine-tuning loss over the trainable parameter set $r \ll \min(m,n)$2, with no added rank regularization term reported.

This absence of an explicit regularizer matters for the LAR interpretation. Many adaptive-rank methods allocate rank under a fixed global budget and therefore require explicit scoring, pruning, or budget redistribution. LR-LoRA instead leaves the nominal base factorization rank $r \ll \min(m,n)$3 fixed as an outer cap on the latent factors $r \ll \min(m,n)$4, but the effective rank of the realized update can be smaller or larger than the nominal $r \ll \min(m,n)$5 after the nonlinear mapping. The paper says this directly: “The learned adaptation dimensionality of each layer can therefore be smaller or larger than the nominal $r \ll \min(m,n)$6.” So the model is adaptive without a hard budget-allocation optimization problem.

Empirically, the method strongly supports the local-adaptivity story. The paper repeatedly shows that different layers learn different stable ranks, and that attention and MLP modules follow systematically different patterns. In the main text, the stable-rank profiles plotted across depth for Qwen2-0.5B at $r \ll \min(m,n)$7 show that update complexity is depth-dependent and differs between MLP and attention projections. In the quantitative stable-rank statistics for Qwen2-0.5B at $r \ll \min(m,n)$8, the mean $r \ll \min(m,n)$9 per-layer standard deviation across adapted modules is:

• LoRA: attention $r(m+n)$0, MLP $r(m+n)$1
• SineLoRA: attention $r(m+n)$2, MLP $r(m+n)$3
• RandLoRA: attention $r(m+n)$4, MLP $r(m+n)$5
• LR-LoRA: attention $r(m+n)$6, MLP $r(m+n)$7

These numbers are central for mapping to LAR. The mean stable rank is higher for LR-LoRA, but more importantly the variance across layers is much larger. The authors argue that SineLoRA and RandLoRA mostly raise expressivity more uniformly, while LR-LoRA induces heterogeneous per-layer rank allocation. In other words, the distinguishing property is not merely “higher rank” but locally differentiated rank.

The paper also provides direct functional evidence that this locality matters. In a parameter-sharing ablation, progressively tying the sinc parameters across layers degrades performance:

• full per-layer learning: $r(m+n)$8
• share $r(m+n)$9: $\mathrm{rank}(\mathbf{B}\mathbf{A}) \le r.$0
• share $\mathrm{rank}(\mathbf{B}\mathbf{A}) \le r.$1: $\mathrm{rank}(\mathbf{B}\mathbf{A}) \le r.$2
• share $\mathrm{rank}(\mathbf{B}\mathbf{A}) \le r.$3: $\mathrm{rank}(\mathbf{B}\mathbf{A}) \le r.$4
• share all: $\mathrm{rank}(\mathbf{B}\mathbf{A}) \le r.$5

When all transfer-function parameters are tied across layers—so the same nonlinearity is used everywhere—accuracy falls back to the LoRA level. This is one of the clearest pieces of evidence that the benefit is genuinely local, layer-specific adaptivity, not simply “nonlinearity added to LoRA.”

The paper further distinguishes between module types. Restricting LR-LoRA to only attention or only MLP hurts performance relative to adapting both. In the main text, attention-only or MLP-only LR-LoRA loses $\mathrm{rank}(\mathbf{B}\mathbf{A}) \le r.$6 and $\mathrm{rank}(\mathbf{B}\mathbf{A}) \le r.$7 points respectively versus full coverage; relative to matched LoRA at the same placement, the gains are $\mathrm{rank}(\mathbf{B}\mathbf{A}) \le r.$8 for attention and $\mathrm{rank}(\mathbf{B}\mathbf{A}) \le r.$9 for MLP, versus $r$0 combined. This superadditivity suggests that attention and MLP layers have distinct capacity requirements that are not well captured by a single uniform rank. The stable-rank table also indicates that MLP adapters tend to realize somewhat higher stable ranks than attention adapters under LR-LoRA ($r$1 vs $r$2 mean stable rank on Qwen2-0.5B), with both showing much larger variation than baselines.

As implemented, rank learning is independent for each adapted adapter matrix. Adapter placement covers attention $r$3 and MLP $r$4 projections. Each instantiated adapter gets its own $r$5, so the granularity is per projection matrix per layer, not merely per transformer block. The transfer-parameter count is

$r$6

where $r$7 is the set of adapted modules and each $r$8 contributes $r$9 scalars $\phi$0. With the default $\phi$1, the overhead is tiny: for example, Qwen2-0.5B goes from $\phi$2M LoRA parameters to $\phi$3M LR-LoRA parameters (+0.95%), Phi-3-8B from $\phi$4M to $\phi$5M (+0.15%), LLaMA3-8B from $\phi$6M to $\phi$7M (+0.13%), and RoBERTa-base from $\phi$8M to $\phi$9M (+0.70%).

In optimization terms, the defaults are stable and simple. For LLM fine-tuning the paper uses AdamW, 3 epochs, learning rate $\Delta \mathbf{W} = \phi(\mathbf{B}\mathbf{A}),$0, adapter dropout $\Delta \mathbf{W} = \phi(\mathbf{B}\mathbf{A}),$1, and batch sizes 16 / 8 / 4 for Qwen2-0.5B / Phi-3-8B / LLaMA3-8B. The LR-LoRA-specific defaults are: $\Delta \mathbf{W} = \phi(\mathbf{B}\mathbf{A}),$2 The paper reports robustness: jointly perturbing $\Delta \mathbf{W} = \phi(\mathbf{B}\mathbf{A}),$3 by $\Delta \mathbf{W} = \phi(\mathbf{B}\mathbf{A}),$4 changes accuracy by at most $\Delta \mathbf{W} = \phi(\mathbf{B}\mathbf{A}),$5 points. Grid resolution saturates quickly: $\Delta \mathbf{W} = \phi(\mathbf{B}\mathbf{A}),$6 yields $\Delta \mathbf{W} = \phi(\mathbf{B}\mathbf{A}),$7 in the ablation harness. Learning bandwidths matters: fixing $\Delta \mathbf{W} = \phi(\mathbf{B}\mathbf{A}),$8 while learning only $\Delta \mathbf{W} = \phi(\mathbf{B}\mathbf{A}),$9 gives only $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$00 over LoRA, versus $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$01 for full LR-LoRA, so learned $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$02 accounts for about 36% of the total gain. Near-zero amplitude initialization is more stable than random initialization.

There is no final discrete rank extraction step, because the method never converts the learned update into an explicit integer-rank choice. The final deployed adapter is just the merged matrix

$$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$03

precomputed once and folded into $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$04 at inference. Consequently, inference has zero overhead. The paper is explicit that $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$05 is precomputed and merged into $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$06, so runtime is identical to the backbone.

Conceptually, the paper positions LR-LoRA as a better inductive bias because it treats adaptation dimensionality as something that should emerge from optimization rather than be imposed a priori. This can be interpreted in several ways relevant to LAR: as adaptive subspace learning, because the low-rank factors still define a latent basis but the nonlinearity reshapes the realized update subspace; as soft rank selection, because the effective spectrum is learned continuously rather than by discrete pruning; and as a form of implicit model selection, because different layers settle on different update complexities under the same training loss. The paper explicitly compares against adaptive-rank PEFT methods such as AdaLoRA, DyLoRA, and ElaLoRA, noting that those methods still rely on a user-specified global capacity budget that is redistributed during training. LR-LoRA differs by not reallocating from a fixed total rank budget; instead, it lets each module’s effective stable rank arise from its own learned transfer function.

The experimental evidence is extensive. The evaluation spans 7 architectures from 125M to 13B parameters, 19 tasks, and four paradigms: commonsense reasoning, GLUE transfer, vision transfer, and instruction tuning. The main language-model backbones are Qwen2-0.5B, Phi-3-8B, and LLaMA3-8B under 15k and 170k training-example regimes. Commonsense tasks are BoolQ, PIQA, SocialIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, and OpenBookQA. GLUE uses RoBERTa-base and RoBERTa-large on SST-2, MRPC, CoLA, QNLI, RTE, and STS-B. Vision experiments use CLIP ViT-B/32 and DINOv2 ViT-B/14. Instruction tuning fine-tunes on Stanford Alpaca and evaluates with MT-Bench.

The main commonsense table shows LR-LoRA is best in all six backbone/regime settings at $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$07:

• Qwen2-0.5B: 55.88 (15k), 61.05 (170k)
• Phi-3-8B: 83.90 (15k), 87.30 (170k)
• LLaMA3-8B: 85.99 (15k), 88.22 (170k)

Relative to LoRA, gains are:

• Qwen2: $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$08 and $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$09
• Phi-3: $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$10 and $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$11
• LLaMA3: $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$12 and $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$13

Relative to RandLoRA, gains range from $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$14 to $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$15. LR-LoRA also beats AdaLoRA, DyLoRA, and ElaLoRA in all entries of that table. On GLUE, LR-LoRA-4 reaches average 84.5 on RoBERTa-base and 88.3 on RoBERTa-large, exceeding the strongest baseline by $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$16 and $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$17 points respectively. On CoLA the gains are especially large: $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$18 on base and $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$19 on large relative to the strongest non-LR-LoRA baseline. On MT-Bench, LR-LoRA scores 4.61 vs LoRA 4.19 on Qwen2-0.5B and 6.55 vs LoRA 6.12 on LLaMA3-8B. On CLIP transfer, the average rises from 81.1 (LoRA) to 82.3; on DINOv2, from 82.0 to 83.2.

A particularly LAR-relevant ablation is the interaction with base rank $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$20. On Qwen2-0.5B (170k), LR-LoRA’s absolute gain over LoRA decreases as $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$21 grows: $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$22 The authors interpret this monotonic pattern as evidence that LR-LoRA is most useful when the fixed-rank bottleneck is binding. That is exactly the regime where adaptive rank allocation should matter most.

So, can LR-LoRA be viewed as an instance of Locally Adaptive Ranking? Yes—conceptually, strongly yes. The mapping is:

• local: each adapted module/layer has its own learned transfer function $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$23, and empirical effective rank varies across depth and between module types;
• adaptive: these transfer functions are learned jointly during training, not preset by heuristic or manually assigned;
• ranking/rank: the method controls low-rank adaptation capacity by changing the realized spectral complexity of $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$24, measured by stable rank.

But there is also an important limitation to this mapping. LR-LoRA does not explicitly learn a discrete rank per layer in the way some LAR formulations might imply. There is no integer $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$25 selected for layer $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$26, no mask over rank components, and no explicit budgeted redistribution of rank units across layers. The nominal LoRA factor rank $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$27 is still chosen by the user, and what is learned is the effective stable rank of the final update, which can exceed or fall below $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$28. Thus, LR-LoRA is best described as a continuous, implicit, per-layer adaptive-rank method, not a discrete rank-allocation method.

Practically, a practitioner should prefer LR-LoRA over fixed-rank LoRA when they suspect that different layers need different adaptation capacities, especially under tight rank budgets. The strongest gains appear at smaller base ranks, on harder reasoning tasks, and across heterogeneous architectures. The trade-offs are favorable: additional parameters are tiny ($$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$29 over LoRA), peak memory rises by at most about $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$30, and throughput drops by at most about $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$31–$$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$32 in reported settings. Inference cost is unchanged after merging. The main complexity cost is implementation: one must support per-adapter transfer functions and the merge-once-per-step training strategy. The paper also notes that LR-LoRA breaks some of LoRA’s backward-pass associativity advantages, because gradients through $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$33 require materializing dense $$\mathbf{W}_{\text{eff}} = \mathbf{W} + \Delta \mathbf{W}, \qquad \Delta \mathbf{W} = \mathbf{B}\mathbf{A},$$34 terms during backprop, though empirically this overhead stays small.

As for caveats, the paper does not provide formal approximation or optimization guarantees for the learned-rank behavior; its evidence is empirical. It also does not provide a discrete rank readout, a direct rank penalty, or a principled stopping criterion for capacity. If someone’s notion of LAR specifically requires explicit per-layer integer-rank assignment under a global budget, LR-LoRA only partially matches that definition. If, however, LAR is understood more broadly as layer-wise adaptive allocation of low-rank adaptation capacity, then LR-LoRA is a clear and strong example of that idea.