Not How Many, But Which: Parameter Placement in Low-Rank Adaptation

Published 12 May 2026 in cs.LG, cs.AI, and cs.CL | (2605.12207v1)

Abstract: We study the \textit{parameter placement problem}: given a fixed budget of $k$ trainable entries within the B matrix of a LoRA adapter (A frozen), does the choice of which $k$ matter? Under supervised fine-tuning, random and informed subsets achieve comparable performance. Under GRPO on base models, random placement fails to improve over the base model, while gradient-informed placement recovers standard LoRA accuracy. This regime dependence traces to gradient structure: SFT gradients are low-rank and directionally stable, so any subset accumulates coherent updates; GRPO gradients are high-rank and near-orthogonal across steps, so only elements with consistently signed gradients retain the learning signal. Our scoring procedure identifies these critical parameters in under 10 seconds at less than 0.5% of training cost. Selected parameters concentrate on residual-stream-writing projections (V, O, Down), stable across model families and scales (1.5B - 8B).

Abstract PDF Upgrade to Chat

Authors (2)

Summary

The paper’s main contribution is reframing LoRA adaptation as a parameter selection challenge rather than merely increasing parameter count.
It introduces a gradient-informed 'circuit discovery' method to efficiently identify a minimal subset of parameters critical for effective adaptation.
The findings demonstrate that while random parameter placement suffices for supervised fine-tuning, informed placement is essential for optimal reinforcement learning performance.

Parameter Placement in Low-Rank Adaptation: An Expert Analysis

The study "Not How Many, But Which: Parameter Placement in Low-Rank Adaptation" (2605.12207) probes the substructure of parameter-efficient fine-tuning (PEFT), reframing LoRA adaptation as a parameter selection challenge. Rather than optimizing for count alone, the authors interrogate which LoRA adapter parameters are trained and how this choice interacts with distinct learning regimes, particularly supervised fine-tuning (SFT) versus reinforcement learning from preferences (GRPO). The core claim is that parameter placement is largely irrelevant for SFT due to high gradient alignment, but is determinative under GRPO, where gradient diversity and instability render most parameter choices ineffective. The authors introduce a lightweight, gradient-informed selection procedure to efficiently isolate the small subset of parameters critical for RL-based adaptation, achieving drastic compression with full performance retention.

Problem Formulation and Theoretical Foundations

Classical LoRA parameterizations typically distribute a fixed parameter budget randomly or uniformly within the $\mathbf{B}$ matrix of the low-rank adapter $\mathbf{BA}$ . The present work formalizes the "parameter placement problem": for a fixed budget $k$ of trainable parameters, does the specific subset within $\mathbf{B}$ matter for downstream performance, and if so, when?

The study motivates exclusive focus on $\mathbf{B}$ —not $\mathbf{A}$ —since with $\mathbf{B}$ initialized at zero, initial updates to $\mathbf{A}$ have no effect due to the outer product structure; empirically, allocating all the trainable budget to $\mathbf{B}$ yields superior optimization efficiency, especially at low budgets.

The hypothesis is grounded in the alignment of learning signals. SFT gradients are shown to be highly coherent and low-rank (as evidenced by high cosine similarity and low effective rank), such that any randomly chosen subset of parameters will generally intercept the dominant update direction. In contrast, GRPO produces gradients with high effective rank and low stepwise alignment; only parameters with stably signed, high-magnitude gradients yield coherent accumulated updates—rendering parameter choice critical.

Gradient-Informed Parameter Selection

The gradient-based selection method, termed "circuit discovery," leverages short scoring runs at initialization (typical $N=50$ forward-backward passes) to aggregate per-parameter statistics. Two scoring functions are considered:

Gradient Magnitude $\mathbf{BA}$ 0: Mean absolute gradient across samples, capturing persistently aligned directions.
Empirical Fisher $\mathbf{BA}$ 1: Mean squared gradient, capturing both mean and variance.

For both scores, the heavy-tailed distribution ensures that a small set of parameters dominates the available signal. At low $\mathbf{BA}$ 2, the rankings are highly stable, and most of the signal is concentrated in a small circuit of the full space.

Experimental Results and Empirical Claims

Extensive ablations across model families (Qwen2.5, Llama-3.x, including vision-language variants) and parameter budgets establish several strong, quantitative results:

SFT Regime: Both random and informed placement of $\mathbf{BA}$ 3 nonzero parameters in LoRA adapters produce nearly identical downstream performance across benchmarks. For example, on Alpaca, Qwen2.5-3B ( $\mathbf{BA}$ 4k) yields 65.8% (random), 66.2% ( $\mathbf{BA}$ 5), and 66.7% (full LoRA) average accuracy; the difference is within noise.
GRPO Regime: Random placement fails entirely; models remain at (or degrade to) base accuracy. Gradient-informed placement matches or surpasses full LoRA at compression ratios $\mathbf{BA}$ 6. E.g., on GSM8K, Qwen2.5-7B with $\mathbf{BA}$ 7k and circuit placement yields 85.2% ( $\mathbf{BA}$ 8), 86.3% ( $\mathbf{BA}$ 9), while random placement stalls at 79.6% (base is 79.3%).

Figure 2: Score distributions for Qwen2.5-3B on GSM8K (left) and MATH (right), revealing that both $k$ 0 and $k$ 1 highlight a heavy-tailed structure—critical for subset selection under RL fine-tuning.

This regime dependency is quantitatively matched by gradient metrics:

Under SFT, effective gradient rank is $k$ 2, with cosine similarity between steps $k$ 3; over 80% of gradient mass accumulates coherently.
Under GRPO, effective rank is high and per-step gradients are near-orthogonal (cosine $k$ 4); less than 25% of gradient energy survives accumulation after many steps (see Table 1 in the main text).

Synthetic experiments further validate the claim: in dense-signal regimes (rank-2 additive residual), random and informed placement converge similarly, while in sparse-signal scenarios (5% high-amplitude entries), only informed placement converges, and random updates fail even at high parameter budgets.

Circuit Structure and Functional Anatomy

Placement analysis reveals that the highest-scoring parameters concentrate non-uniformly over depth and module type:

Module Preference: Circuit selection favors residual-stream readers and writers—specifically, the V (value), O (output), and Down projections.
Layer Density: Top-circuit elements cluster in early and mid-depth layers, with slight variation across architectures.
Figure 4: Circuit distribution in Llama-3.2-3B-Instruct on GSM8K (top 500K). While large-budget circuits are more distributed, V/O modules in early layers are highly represented. Instruct-tuned models parallel base Qwen2.5 patterns.

Figure 6: For Qwen2.5-VL-7B on MathVista, discovered circuits automatically assign budget to the vision encoder, indicating adaptation bottlenecks in the multimodal pathway.

Functional Characterization

Interrogation of circuit modules via gradient sign consistency and SVD alignment yields two broad mechanistic signatures:

Attention Readers (V, K): Maximal alignment with top singular vectors of the pretrained weights, indicating their role in modulating input selection.
Residual Writers (O, Down): Spectral concentration, i.e., updates are low-rank and correspond to direct modifications to the core residual stream, aligning with their functional effect.
Figure 8: Gradient sign consistency by module, showing that only modules overrepresented in circuits have persistently aligned gradients across steps.

Figure 10: SVD alignment analysis stratifies modules into readers (high left alignment) and writers (high spectral concentration), recapitulating observed circuit concentration.

Causal Analysis via Knockout Experiments

Zeroing parameters in the trained adapter by rank-ordered circuit scores versus at random demonstrates the causal leverage of circuit elements. Ordered knockout (by $k$ 5) rapidly degrades accuracy, with up to 12.2 percentage point drops compared to random removal at the same budget fraction—a direct demonstration that gradient-based scores select functionally essential parameters.

Figure 12: Accuracy drop as a function of fraction of $k$ 6 entries zeroed out by $k$ 7 score, confirming that critical parameters are concentrated in the identified circuit.

Behavioral Impact and Dynamic Trajectories

Displacement of circuit parameters, as measured by effective update norm, is tightly coupled with behavioral change as seen in output token distributions. Despite similar update magnitudes, randomly placed parameters fail to induce logit-level changes; selected circuits cause significantly greater shifts in model predictions per parameter changed. Moreover, most divergence occurs in later layers, despite circuit density being highest earlier—a phenomenon consistent with downstream amplification through the residual stream.

Theoretical and Practical Implications

The findings fundamentally refine the community’s understanding of PEFT:

LoRA parameter count alone does not suffice for RL-based adaptation; only a functionally targeted selection harnesses the highly distributed, unstable RL learning signal.
A sub-0.1% parameter mask suffices, with informed selection, for full RL performance recovery in large models (up to 8B parameters)—enabling significant memory and compute savings for RLHF and related tasks.
Gradient-based circuit discovery is extremely efficient: $k$ 8 forward-backward passes at initialization suffice; cost is negligible compared to training, and can be amortized or reused across multiple runs.
Selected circuits generalize across distributions: Circuits discovered for one dataset or task often overlap substantially ( $k$ 9 chance) with those for related tasks, supporting implicit regularization and improved OOD generalization compared to dense adaptation.

Theoretically, this extends findings on the intrinsic dimensionality of adaptation and supports the view of adaptation as a form of elicited subnetwork transfer, rather than distributed capacity expansion [aghajanyan2021intrinsic]. Practically, the work motivates new PEFT pipelines for RL that always precompute and deploy parameter selection per task or intended reward function—potentially as a plug-in for future LoRA-based alignment frameworks.

Conclusion

This study demonstrates that performance in LoRA-based PEFT is not simply determined by parameter count but critically depends on strategic placement, especially under RL-based objectives with ill-conditioned, high-variance gradients. Random parameter placement suffices for SFT but fails for RL. The lightweight, gradient-based circuit discovery method identifies the minimal sufficient subnetwork for effective adaptation. The practical implications include radical reductions in both training time and memory footprint for RL-based fine-tuning, advancing the state-of-the-art in parameter-efficient adaptation. Future research may explore compositional or hierarchical circuit discovery, further reductions in budget, or automated selection for mixed-modality models.