Key-Value Low-Rank Adaptation

• Key-Value Low-Rank Adaptation is a technique that modifies transformer key and value projections using low-rank updates to capture essential information in a reduced subspace.
• It employs various methods such as adaptive rank allocation, spectral reparameterizations, and optimizer-aware updates to enhance model stability and performance.
• This approach facilitates efficient transformer fine-tuning by enabling shared basis components, cross-layer adaptations, and tensorized factorization of attention mechanisms.

Key-Value Low-Rank Adaptation denotes a family of transformer adaptation techniques that modify the key and value projections with low-rank structure rather than full dense updates. In the standard LoRA-style PEFT setting, a frozen attention weight is replaced by $W' = W + \Delta W$, with $\Delta W$ parameterized as a low-rank product; in this usage, “Key-Value Low-Rank Adaptation” usually means adding such adapters to the key and value matrices $W_k$ and $W_v$ in each layer independently (Zhong et al., 2024). The same label is also used more broadly for methods that vary rank across layers or modules, share low-rank components across depth or across $Q/K/V$, use tensorized or routed compositions, reparameterize updates spectrally, or alter the K/V parameterization itself during pretraining to reduce KV-cache cost (O'Neill et al., 16 Jan 2026).

1. Canonical formulation and transformer placement

The canonical formulation inherits directly from LoRA. For a linear weight matrix $\mathbf{W} \in \mathbb{R}^{m \times n}$, the adapted weight is written as

$\mathbf{W}_\text{eff} = \mathbf{W} + \mathbf{BA},$

with $\mathbf{A} \in \mathbb{R}^{r \times n}$, $\mathbf{B} \in \mathbb{R}^{m \times r}$, and $\mathrm{rank}(\mathbf{BA}) \le r$. An equivalent notation writes

$\Delta W$0

with $\Delta W$1 and $\Delta W$2, where $\Delta W$3 (Zhong et al., 2024). In both notations, the central constraint is that the task-specific update lives in a low-dimensional subspace.

Within attention, the standard decomposition is

$\Delta W$4

Under standard K/V low-rank adaptation, one replaces $\Delta W$5 and $\Delta W$6 by $\Delta W$7 and $\Delta W$8, with each $\Delta W$9 low rank. The standard LoRA interpretation is per-layer and per-projection: each layer $W_k$0 has its own low-rank pair, and the LP/HP or $W_k$1 factors for that layer are paired only with the features of that layer (Zhong et al., 2024).

Placement conventions are not uniform across the literature. One line of work follows what it calls “the standard setting in LoRA” and inserts LoRA only on the query and value projection layers of each multi-head attention module; in that setting, keys and output projections remain frozen, giving 24 LoRA layers for RoBERTa-base and GPT2-medium and 48 for RoBERTa-large (Zhang et al., 2024). Another line applies adapters to all attention and MLP projections—query, key, value, output, plus gate, up, and down—throughout the backbone, while not adapting the final output head (Garg et al., 3 Jun 2026). A comprehensive account of Key-Value Low-Rank Adaptation therefore has to distinguish the core concept—low-rank modification of $W_k$2 and $W_k$3—from the broader design choice of which additional projections are co-adapted.

2. Module-specific behavior in attention blocks

A recurrent result is that attention projections do not exhibit a uniform preference for rank or placement. Methods that explicitly allocate rank across modules report different patterns depending on the task, model, and scoring rule.

ALoRA treats a LLaMA-2-style transformer block as seven distinct modules—Query, Key, Value, Output, Gate, Up, and Down—and reallocates LoRA rank among them under a fixed global budget. Its visualization on E2E reports that more LoRA rank budgets are put to adapt the query and key modules, while the value and output modules in the self-attention are less emphasized; it also reports that the feed-forward layer requires fewer LoRA ranks (Liu et al., 2024). By contrast, GoRA allocates ranks from gradient-derived importance scores and, on LLaMA3.1-8B fine-tuned for MetaMathQA, reports that most ranks are allocated to the $W_k$4 layers, while the $W_k$5 layers receive the fewest rank allocations (He et al., 13 Feb 2025).

This divergence is not merely anecdotal. It indicates that key and value adaptation cannot be reduced to a single universal heuristic such as “values matter most” or “keys matter most.” A plausible implication is that K/V low-rank adaptation is intrinsically task-dependent: for some objectives, adapting attention geometry through $W_k$6 is prioritized, whereas for others, modifying content transport through $W_k$7 dominates. The literature therefore increasingly replaces uniform per-matrix rank choices with allocation mechanisms that respond to validation loss, stable-rank priors, or gradient statistics.

A second module-level observation concerns the sufficiency of K/V-only or attention-only tuning. In LR-LoRA’s component ablation on Qwen2, “Q,K,V Only” is explicitly listed as a configuration and yields accuracy $W_k$8 versus 67.8 for the “no adaptation LoRA-equivalent baseline” in that table, whereas “All Components” reaches 71.4 (Garg et al., 3 Jun 2026). This does not negate the utility of K/V adaptation, but it does show that K/V-only designs can leave performance on the table when output projections or MLP blocks are also strongly task-relevant.

3. Rank, effective capacity, and adaptive allocation

The most persistent criticism of standard LoRA in this area is not merely that the nominal rank $W_k$9 is small, but that a fixed $W_v$0 imposes a fixed low-rank inductive bias on all adapter layers and modules. The literature distinguishes at least three quantities: the algebraic adapter rank $W_v$1, the realized rank of the update matrix, and the stable rank

$W_v$2

which acts as a continuous proxy for effective dimensionality (Garg et al., 3 Jun 2026).

"Learnable Rank LoRA" replaces the fixed-rank update $W_v$3 by $W_v$4, where $W_v$5 is a learned elementwise nonlinearity expressed as a sum of shifted sinc basis functions. Because $W_v$6 is not itself constrained to factor as $W_v$7, its stable rank can differ substantially from the nominal factorization rank. On Qwen2-0.5B at nominal $W_v$8, it reports stable rank $W_v$9 for LoRA attention updates and $Q/K/V$0 for LR-LoRA attention updates, with analogous increases for MLP layers (Garg et al., 3 Jun 2026). The same paper also shows depth-dependent stable-rank profiles in which early layers tend to lower stable rank and later layers higher, while attention and MLP layers exhibit distinct depth profiles.

AutoLoRA attacks the same problem by decomposing each low-rank update into explicit rank-1 terms,

$Q/K/V$1

with $Q/K/V$2 obtained by a softmax over trainable variables $Q/K/V$3. The effective rank of layer $Q/K/V$4 is then determined by thresholding the learned $Q/K/V$5 at $Q/K/V$6, after a bi-level meta-learning loop that updates LoRA weights on training loss and selection variables on validation loss (Zhang et al., 2024). In its experiments, this machinery is applied to query and value LoRA modules, and Figure 1 shows that different transformer layers receive different ranks and that Q and V layers need not share the same rank profile.

SR-LoRA substitutes a model prior for explicit search. It computes the stable rank of each pretrained weight,

$Q/K/V$7

and allocates adapter rank from that quantity, arguing that stable rank reflects the intrinsic dimensionality of the pretrained transformation. In its ViT-based experiments, this stable-rank-guided allocation outperforms fixed-rank LoRA and several adaptive baselines in few-shot, large-gap regimes; for MedFM 1-shot mean AUC, SR-LoRA reports 72.47 versus 64.09 for LoRA (Zhang et al., 30 Jun 2025). Although its reported placements are $Q/K/V$8, $Q/K/V$9, and $\mathbf{W} \in \mathbb{R}^{m \times n}$0, the construction applies equally to $\mathbf{W} \in \mathbb{R}^{m \times n}$1.

PoLAR exposes a more severe issue: the nominal rank supplied to LoRA can be largely unused. On Llama-2-7B with nominal rank $\mathbf{W} \in \mathbb{R}^{m \times n}$2, it reports many learned LoRA updates with stable rank near 1, in some cases as low as $\mathbf{W} \in \mathbb{R}^{m \times n}$3, and interprets the resulting geometry as directional diversity collapse (Lion et al., 3 Jun 2025). This result is important for K/V adaptation because it shows that the effective expressiveness of a K/V adapter can be far below its algebraic rank unless the parameterization or optimizer actively prevents collapse.

4. Cross-layer, cross-projection, and shared K/V structures

Beyond per-layer independent adapters, a substantial body of work introduces structured sharing across layers, across $\mathbf{W} \in \mathbb{R}^{m \times n}$4, or across heads.

Method	K/V structure	Representative result
Lily	Local LPs $\mathbf{W} \in \mathbb{R}^{m \times n}$5, global HP experts $\mathbf{W} \in \mathbb{R}^{m \times n}$6, soft router $\mathbf{W} \in \mathbb{R}^{m \times n}$7	On ViT-B/16, Lily (kvffn) reaches 77.3 on VTAB-1K with 0.76M params, versus LoRA 76.4 with 1.17M
TensLoRA	Tucker factorization of QKV or QKV_Depth tensors	Under isorank, QKV uses 33.7% of LoRA parameters on ViT and achieves $\mathbf{W} \in \mathbb{R}^{m \times n}$8 on EuroSAT versus $\mathbf{W} \in \mathbb{R}^{m \times n}$9 for Att
LRKV	Shared full-rank $\mathbf{W}_\text{eff} = \mathbf{W} + \mathbf{BA},$0 plus head-specific low-rank residuals $\mathbf{W}_\text{eff} = \mathbf{W} + \mathbf{BA},$1	At 2.5B, LRKV with $\mathbf{W}_\text{eff} = \mathbf{W} + \mathbf{BA},$2 uses 52.6% KV cache and reaches equivalent model quality with up to 20–25% less training compute

Lily generalizes LoRA across depth by keeping layer-specific low-dimension projectors $\mathbf{W}_\text{eff} = \mathbf{W} + \mathbf{BA},$3 while sharing a pool of high-dimension projector experts $\mathbf{W}_\text{eff} = \mathbf{W} + \mathbf{BA},$4 across all layers. A router maps the low-dimensional features to expert weights, producing a combined high-dimension projector

$\mathbf{W}_\text{eff} = \mathbf{W} + \mathbf{BA},$5

When applied to attention, this yields interconnected K/V updates in which the HP side is shared across layers and recombined through data-dependent routing rather than learned independently per layer (Zhong et al., 2024). The paper explicitly identifies “Lily (kvffn)” as the configuration most closely aligned with Key-Value Low-Rank Adaptation.

TensLoRA moves from matrices to tensors. It stacks LoRA updates into tensors such as $\mathbf{W}_\text{eff} = \mathbf{W} + \mathbf{BA},$6 for $\mathbf{W}_\text{eff} = \mathbf{W} + \mathbf{BA},$7 within one layer or $\mathbf{W}_\text{eff} = \mathbf{W} + \mathbf{BA},$8 for QKV across depth, and then constrains this tensor to a Tucker low-rank manifold (Marmoret et al., 22 Sep 2025). The QKV mode factor $\mathbf{W}_\text{eff} = \mathbf{W} + \mathbf{BA},$9 explicitly couples Q, K, and V, while the depth factor $\mathbf{A} \in \mathbb{R}^{r \times n}$0 couples layers. This replaces independent per-projection adapters with a shared projection-type subspace and a shared depth subspace.

LRKV uses the phrase “low-rank KV adaptation” in a different but related sense. It is not a PEFT method over a frozen backbone but a replacement for standard multi-head attention during pretraining. Each head’s key and value projections are decomposed as

$\mathbf{A} \in \mathbb{R}^{r \times n}$1

with a shared full-rank base and head-specific low-rank residuals (O'Neill et al., 16 Jan 2026). This gives a continuous trade-off between complete sharing ($\mathbf{A} \in \mathbb{R}^{r \times n}$2, equivalent to MQA) and fully independent heads, while reducing KV-cache memory by exploiting redundancy across heads.

A more indirect extension is MoR. Its experiments replace only FFN linear layers with a mixture of transformed low-rank directions, but the formulation is generic: $\mathbf{A} \in \mathbb{R}^{r \times n}$3 The paper states that the same machinery could be applied to $\mathbf{A} \in \mathbb{R}^{r \times n}$4 and $\mathbf{A} \in \mathbb{R}^{r \times n}$5, yielding a task- and input-dependent mixture of low-rank key/value adapters (Tang et al., 2024). This extends the broader trend from fixed per-layer K/V updates toward shared bases and routed combinations.

5. Spectral, geometric, and optimizer-aware parameterizations

Another major axis of development reparameterizes the low-rank update itself rather than merely reallocating rank. These methods are especially relevant for K/V matrices because they target stability, spectrum preservation, or closer emulation of full fine-tuning.

DuDe combines magnitude–direction decomposition with SVD-based initialization. For a pretrained weight $\mathbf{A} \in \mathbb{R}^{r \times n}$6, it extracts the top-$\mathbf{A} \in \mathbb{R}^{r \times n}$7 singular component $\mathbf{A} \in \mathbb{R}^{r \times n}$8, freezes the residual $\mathbf{A} \in \mathbb{R}^{r \times n}$9, initializes

$\mathbf{B} \in \mathbb{R}^{m \times r}$0

and adapts

$\mathbf{B} \in \mathbb{R}^{m \times r}$1

Its main instantiation is on $\mathbf{B} \in \mathbb{R}^{m \times r}$2 and $\mathbf{B} \in \mathbb{R}^{m \times r}$3, with a combined $\mathbf{B} \in \mathbb{R}^{m \times r}$4 case for Phi-4 small; on LLaMA2-13B, it reports average commonsense accuracy 63.74 versus 61.51 for LoRA and a HellaSwag improvement of +11.76 points (Han et al., 20 May 2025). Although the core experiments are query/value rather than key/value, the paper presents the construction as directly transferable to $\mathbf{B} \in \mathbb{R}^{m \times r}$5 and $\mathbf{B} \in \mathbb{R}^{m \times r}$6.

SALT operates in the singular-value basis. It performs SVD once per frozen weight, adapts the top $\mathbf{B} \in \mathbb{R}^{m \times r}$7 singular values through trainable scale and shift, and applies a low-rank correction in the residual singular-value subspace: $\mathbf{B} \in \mathbb{R}^{m \times r}$8 The updated matrix is then reconstructed as

$\mathbf{B} \in \mathbb{R}^{m \times r}$9

On the SAM encoder, SALT rank-256 uses 3.9% trainable parameters and reports average Dice 0.74, compared with 0.70 for LoRA rank-256 at 14.08% trainable parameters (Elsayed et al., 20 Mar 2025). The paper explicitly proposes a SALT-style adaptation for $\mathrm{rank}(\mathbf{BA}) \le r$0 and $\mathrm{rank}(\mathbf{BA}) \le r$1 in transformers.

PoLAR replaces the standard LoRA update $\mathrm{rank}(\mathbf{BA}) \le r$2 by

$\mathrm{rank}(\mathbf{BA}) \le r$3

with $\mathrm{rank}(\mathbf{BA}) \le r$4, $\mathrm{rank}(\mathbf{BA}) \le r$5, and unconstrained $\mathrm{rank}(\mathbf{BA}) \le r$6, thereby separating orthonormal direction bases from scale and mixing (Lion et al., 3 Jun 2025). The paper argues that this counters LoRA’s low stable-rank pathology and proves an exponentially faster convergence rate on a canonical low-rank adaptation problem. On Llama-2-7B, PoLAR at rank 32 reports average commonsense accuracy 73.71 versus 72.18 for LoRA and 72.47 for DoRA.

LoFT keeps the conventional low-rank parameterization $\mathrm{rank}(\mathbf{BA}) \le r$7 but changes the optimizer geometry. It alternates updates of $\mathrm{rank}(\mathbf{BA}) \le r$8 and $\mathrm{rank}(\mathbf{BA}) \le r$9, scales gradients by $\Delta W$00 or $\Delta W$01, and calibrates Adam’s first and second moments so that low-rank updates track full-model AdamW updates projected into the same subspace. The resulting projected gradient step is

$\Delta W$02

Applied to transformer attention $\Delta W$03 and FFN matrices, LoFT reports, on LLaMA-7B at rank 16, average commonsense accuracy 76.08 versus 73.57 for LoRA (Tastan et al., 27 May 2025). Its relevance to K/V adaptation is direct: one may adapt only $\Delta W$04 and $\Delta W$05 while preserving optimizer dynamics closer to full fine-tuning within the chosen low-rank subspaces.

6. Empirical regularities, misconceptions, and open problems

Several empirical regularities recur across these papers. First, key/value low-rank adaptation is effective, but it is not a guarantee that K/V-only placement is sufficient. LR-LoRA’s single-seed ablation shows that “Q,K,V Only” yields only a marginal change relative to the LoRA-equivalent baseline in that restricted setup, whereas “All Components” gives the best result; at the same time, its broader experiments show that adapting attention projections can improve structured tasks more than FFN-only adaptation (Garg et al., 3 Jun 2026). The literature therefore supports K/V adaptation as an important component, not as a universally complete substitute for broader attention or MLP adaptation.

Second, nominal rank and useful rank should not be conflated. PoLAR’s stable-rank analysis shows that a nominal rank-32 LoRA update can behave almost like a rank-1 perturbation (Lion et al., 3 Jun 2025). This undercuts the common assumption that simply increasing $\Delta W$06 reliably increases functional capacity. A related architectural result appears in LRKV: reducing K/V redundancy across heads need not degrade quality if the parameterization preserves enough head diversity; the paper’s operator-space analysis finds that LRKV with $\Delta W$07 preserves nearly all functional head diversity relative to standard multi-head attention (O'Neill et al., 16 Jan 2026).

Third, the term itself is overloaded. In PEFT, Key-Value Low-Rank Adaptation usually denotes low-rank updates attached to pretrained $\Delta W$08 and $\Delta W$09. In LRKV, it denotes a pretraining-time attention mechanism with a shared full-rank K/V basis and low-rank head-specific residuals (O'Neill et al., 16 Jan 2026). The conceptual overlap is real—both exploit low-rank structure in key/value projections—but the training regimes, objectives, and systems consequences differ.

Open questions remain substantial. Lily does not provide an explicit rank-theoretic analysis of how shared HP experts, layer-varying LPs, and routing change the span of achievable $\Delta W$10 for K/V, and it leaves language-model scaling to future work (Zhong et al., 2024). LR-LoRA reports attention-level stable-rank profiles but does not provide separate stable-rank curves for Q, K, V, and O (Garg et al., 3 Jun 2026). LRKV identifies adaptive rank allocation across layers or heads, long-context behavior, and combinations with GQA or FlashAttention-3 as future directions (O'Neill et al., 16 Jan 2026). Across the broader literature, a consistent unresolved issue is how to allocate K/V rank across depth, projection type, and heads without turning PEFT into an expensive search procedure.

Taken together, the field has moved from the original per-layer, independent low-rank perturbation of $\Delta W$11 and $\Delta W$12 toward a richer design space. Contemporary Key-Value Low-Rank Adaptation now encompasses fixed-rank adapters, learned effective rank, stable-rank-guided allocation, cross-layer expert sharing, tensor factorization across $\Delta W$13 and depth, spectral parameterizations, geometry-aware updates, and pretraining-time K/V reparameterizations for KV-cache efficiency. The unifying premise is unchanged: transformer attention contains substantial low-dimensional structure in the key and value pathways, but exploiting that structure well requires careful choices about where low rank is imposed, how rank is distributed, and what notion of rank is actually being optimized.