KRAdapter: Efficient High-Rank PEFT

• KRAdapter is a parameter-efficient fine-tuning method that uses the Khatri–Rao product to create high effective-rank updates for complex, high-frequency data.
• It improves out-of-distribution generalization and robustness in both vision-language models and large language models while retaining efficient memory and compute usage.
• Empirical evaluations show KRAdapter achieves flatter singular value spectra and lower nuclear-norm errors compared to traditional low-rank methods like LoRA.

KRAdapter is a parameter-efficient fine-tuning (PEFT) algorithm designed to upgrade the representational capacity of weight updates in large pre-trained neural networks, particularly in scenarios where low-rank adaptation methods like LoRA are insufficient, such as when modeling data with high effective rank or intricate spectral properties. By leveraging the Khatri–Rao product—a column-wise Kronecker product—KRAdapter increases the effective rank of learned updates while retaining the practical memory and compute profiles central to state-of-the-art PEFT approaches. KRAdapter demonstrates performance gains on both vision-LLMs and LLMs, with particular strength in out-of-distribution (OOD) generalization, and maintains computational efficiency compatible with billion-scale neural architectures (Albert et al., 1 Aug 2025).

1. Parameter-efficient Fine-tuning Formulation

In the canonical PEFT setting, one begins with a pre-trained weight matrix $$W_0 \in \mathbb{R}^{d_\text{out} \times d_\text{in}}$$. Fine-tuning introduces a small trainable update $\Delta W$ so that for an input $x \in \mathbb{R}^{d_\text{in}}$, the model computes

$h = (W_0 + \Delta W)x.$

Full fine-tuning makes all $d_\text{out} \times d_\text{in}$ entries of $\Delta W$ trainable, whereas LoRA restricts $\Delta W$ to be rank-$r$:

$$\Delta W = B A,\quad A \in \mathbb{R}^{r \times d_\text{in}},\; B \in \mathbb{R}^{d_\text{out} \times r},$$

training only $A$ and $\Delta W$0, typically with $\Delta W$1. The limitation of LoRA arises when $\Delta W$2 must approximate full-rank, high-frequency, or high effective rank matrices, a situation common in multi-modal and OOD tasks.

2. Mathematical Construction of KRAdapter

KRAdapter parameterizes updates via the Khatri–Rao product. Let $\Delta W$3 and $\Delta W$4 be trainable matrices, with $\Delta W$5. The Khatri–Rao product ($\Delta W$6) is defined column-wise: for each $\Delta W$7,

$\Delta W$8

with $\Delta W$9 and $x \in \mathbb{R}^{d_\text{in}}$0 denoting the $x \in \mathbb{R}^{d_\text{in}}$1th columns of $x \in \mathbb{R}^{d_\text{in}}$2 and $x \in \mathbb{R}^{d_\text{in}}$3. Stacking these for all $x \in \mathbb{R}^{d_\text{in}}$4,

$x \in \mathbb{R}^{d_\text{in}}$5

The update is then constructed as:

$x \in \mathbb{R}^{d_\text{in}}$6

truncating as needed. A scalar $x \in \mathbb{R}^{d_\text{in}}$7 (e.g., $x \in \mathbb{R}^{d_\text{in}}$8 for vision models) scales the update, and the forward pass is $x \in \mathbb{R}^{d_\text{in}}$9.

This formulation, by construction, produces an update with high effective rank:

• With random (i.i.d.) $h = (W_0 + \Delta W)x.$0, the columns of $h = (W_0 + \Delta W)x.$1 are almost surely linearly independent if $h = (W_0 + \Delta W)x.$2, $h = (W_0 + \Delta W)x.$3.
• Empirically, $h = (W_0 + \Delta W)x.$4 yields a much flatter singular value spectrum than LoRA or Kronecker-product adapters.

3. Spectral Properties and Effective Rank

Low-rank LoRA updates have singular values dropping sharply to zero after the $h = (W_0 + \Delta W)x.$5 component, limiting their expressivity for high-rank matrix approximation.

KRAdapter, in contrast, delivers updates with near-full rank and slow spectral decay. Effective rank, defined as

$h = (W_0 + \Delta W)x.$6

with $h = (W_0 + \Delta W)x.$7 the singular values of $h = (W_0 + \Delta W)x.$8, is consistently higher with KRAdapter than LoRA, SinLoRA, RandLoRA, or Kronecker adapters (Albert et al., 1 Aug 2025). Synthetic benchmarks with diverse spectra (random Gaussian, PCA-whitened, high/low-frequency sinusoids, CLIP weight-deltas) confirm that KRAdapter matches LoRA on strictly low-rank targets but substantially outperforms on high-rank and high-frequency scenarios.

4. Computational Efficiency and Implementation

KRAdapter is designed to match or minimally exceed the compute and memory profiles of LoRA:

• Number of parameters: $h = (W_0 + \Delta W)x.$9, minimized for $d_\text{out} \times d_\text{in}$0.
• For $d_\text{out} \times d_\text{in}$1, $d_\text{out} \times d_\text{in}$2.
• LoRA with rank $d_\text{out} \times d_\text{in}$3 needs $d_\text{out} \times d_\text{in}$4, commonly $d_\text{out} \times d_\text{in}$5 for $d_\text{out} \times d_\text{in}$6.
• Extra FLOPs for forward pass is one $d_\text{out} \times d_\text{in}$7 matrix–vector multiplication, negligible versus the cost of $d_\text{out} \times d_\text{in}$8 ($d_\text{out} \times d_\text{in}$9 ms on 1B-parameter models).
• Training speed and VRAM usage are within $\Delta W$0–$\Delta W$1 of LoRA.

The update is efficiently realized by reshaping and stacking columns, exploiting Khatri–Rao structure for high throughput.

5. Empirical Evaluation and Benchmarks

KRAdapter has been extensively benchmarked:

Synthetic Matrix Approximation

• Benchmarks use matrices with controlled spectral profiles (Gaussian, sparse, decorrelated, low-rank, CLIP-deltas, superposed sinusoids).
• KRAdapter uniformly outperforms LoRA except on strictly low-rank cases and provides the flattest spectrum approximation (lowest nuclear-norm error relative to LoRA).

Vision-LLMs

• Fine-tuned on CLIP variants (ViT-B/32, ViT-L/14, ViT-H/14) across 11 few-shot datasets, ImageNet (50%/100%), and VTAB1k (Natural, Structured, Specialized).
• On 11 classical vision tasks, KRAdapter exceeds LoRA and other adapters by $\Delta W$2–$\Delta W$3.
• For out-of-distribution (OOD) robustness, the generalization ratio $\Delta W$4 is $\Delta W$5 for KRAdapter on ViT-B/32, compared to $\Delta W$6 for LoRA.
• KRAdapter’s updates show smaller nuclear/Frobenius norm shifts from zero-shot, correlating with greater robustness.

LLMs

• Applied to Llama3.1-8B and Qwen2.5-7B (adapters on key/value projections).
• Trained on science QA datasets (SIQA, ARC-E, ARC-C, OBQA), with evaluation on in-distribution, near-distribution (HellaSwag), and OOD (BoolQ, PIQA, WinoGrande).
• KRAdapter achieves the highest average OOD scores: e.g., Llama3 OOD $\Delta W$7 for KRAdapter versus $\Delta W$8 for LoRA, $\Delta W$9 for KronA.

6. Hyperparameters, Practical Use, and Limitations

Typical hyperparameters:

• $\Delta W$0 (trade-off parameter count vs approximation).
• Scaling $\Delta W$1 (vision) or $\Delta W$2 with reweighting for LLMs.
• Learning rates: $\Delta W$3 (matrix toy), $\Delta W$4 (CLIP/LLM), AdamW optimizer.
• Dropout $\Delta W$5 (LLMs) to regularize adapters.

KRAdapter’s minimum parameter budget is $\Delta W$6, which can exceed LoRA with extreme rank-$\Delta W$7 ($\Delta W$8) setups. For scenarios requiring tight rank constraints or extreme compactness, LoRA may be preferred. KRAdapter is suboptimal on strictly low-rank targets and, in some cases, full-rank random parametrizations match or slightly exceed its in-distribution performance on large models.

Future research directions include deploying nested low-rank decompositions for $\Delta W$9 and $r$0, formal study of Khatri–Rao spectral shaping under realistic initializations, and extension to convolutional or other structured layers.

7. Significance and Broader Applicability

KRAdapter advances PEFT by enabling high effective-rank updates while efficiently utilizing parameters and compute. Its spectral properties enhance generalization, especially for tasks involving distribution shift (OOD), multi-modal, and compositional learning. Unlike strictly low-rank approaches, KRAdapter’s update parametrization, via the Khatri–Rao product, offers a theoretically and empirically justified trade-off between resource footprint and expressive adaptation. Empirical studies demonstrate superiority over LoRA and alternative adapters on both vision-language and LLM benchmarks, with particular gains in robustness and OOD accuracy. This approach is particularly relevant for evolving PEFT requirements as models and deployment contexts diversify (Albert et al., 1 Aug 2025).