Conv-LoRA: Convolutional Low-Rank Adaptation

• Conv-LoRA is an extension of low-rank adaptation that models convolutional kernel updates as additive low-rank operators for efficient adaptation.
• It implements the update via parallel convolutions, merging the low-rank update seamlessly into the frozen pretrained kernel.
• The approach maintains original task loss while decoupling forward expressivity from gradient scaling issues, improving convergence and inference efficiency.

Convolutional Low-Rank Adaptation (Conv-LoRA) is the extension of Low-Rank Adaptation to convolutional kernels by representing the trainable update to a frozen pretrained kernel as a low-rank operator. In the LoRA formulation, a pretrained weight $W_0$ is kept fixed and the effective weight becomes $W_0 + \Delta W$, with $\Delta W = BA$ or, in the scaled form, $\Delta W = \frac{\alpha}{r}BA$; for convolution, the kernel is either flattened to a matrix $$W_{0,\mathrm{mat}} \in \mathbb{R}^{C_{\text{out}} \times (C_{\text{in}}k_hk_w)}$$ and adapted in that space, or realized as two convolutions in parallel whose composition is equivalent to a low-rank kernel update. This preserves the original task loss, restricts optimization to a low-dimensional parameter subspace, and allows the learned update to be merged back into the base kernel for inference. A later geometric analysis of low-rank adaptation further shows that the standard LoRA parameterization induces anisotropic gradient scaling through the singular values of the low-rank factors, and proposes SDS-LoRA as a structurally decoupled parameterization whose rationale transfers directly to convolutional adaptation (Hu et al., 2021, Oh et al., 15 Jun 2026).

1. Formalization of convolutional low-rank updates

The canonical LoRA construction begins with a linear weight $W_0 \in \mathbb{R}^{d \times k}$ and replaces full fine-tuning of $\Delta W \in \mathbb{R}^{d \times k}$ by a low-rank parameterization

$W_0 + \Delta W = W_0 + BA,$

with

$$B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k}, \qquad r \ll \min(d,k).$$

Only $A$ and $W_0 + \Delta W$0 are trainable, and the forward map for input $W_0 + \Delta W$1 becomes

$W_0 + \Delta W$2

or, in the scaled form used to stabilize training across ranks,

$W_0 + \Delta W$3

The task objective is unchanged; only the parameterization of the update is altered (Hu et al., 2021).

For convolutional layers, the same construction is obtained by viewing the kernel as a matrix-valued linear operator. A kernel

$W_0 + \Delta W$4

is flattened to

$W_0 + \Delta W$5

Conv-LoRA then applies

$W_0 + \Delta W$6

and reshapes $W_0 + \Delta W$7 back to the original kernel shape. In im2col form, the output can be written as

$W_0 + \Delta W$8

This matrixification is not a separate objective or a separate optimization problem; it is the same additive low-rank reparameterization applied to a convolutional operator. The original LoRA paper states that “the principles are generally applicable to any neural networks with dense layers,” and the convolutional extension follows by treating each local patch-to-output-channel map as a matrix multiplication. This suggests that Conv-LoRA is best understood as the convolutional instantiation of a general low-rank update formalism rather than as an unrelated PEFT method.

2. Realization as parallel convolutions and merged deployment

The most direct implementation of Conv-LoRA realizes the low-rank update as two convolutions. The first convolution corresponds to $W_0 + \Delta W$9: it maps the input patch space to rank $\Delta W = BA$0 using weights of shape

$\Delta W = BA$1

The second corresponds to $\Delta W = BA$2: it expands from rank $\Delta W = BA$3 to $\Delta W = BA$4 through a $\Delta W = BA$5 convolution with weights of shape

$\Delta W = BA$6

The resulting output is

$\Delta W = BA$7

Per convolutional layer, the trainable parameter count is therefore

$\Delta W = BA$8

rather than the full

$\Delta W = BA$9

This is the exact convolutional analogue of the LoRA parameter count $\Delta W = \frac{\alpha}{r}BA$0 for dense layers (Hu et al., 2021).

Initialization follows the same zero-start principle as in the original LoRA formulation: $\Delta W = \frac{\alpha}{r}BA$1 is randomly initialized with a Gaussian distribution, $\Delta W = \frac{\alpha}{r}BA$2 is initialized to zero, and consequently $\Delta W = \frac{\alpha}{r}BA$3 at initialization. The adapted model therefore behaves exactly like the pretrained model at the first forward pass. In convolutional form, this means that the low-rank branch initially contributes zero activation, avoiding a cold-start perturbation of pretrained filters.

A defining systems property of LoRA is that the update is additive and linear, so it can be merged into the frozen weight after training. For Conv-LoRA, if $\Delta W = \frac{\alpha}{r}BA$4 denotes the weights of the first convolution and $\Delta W = \frac{\alpha}{r}BA$5 the weights of the $\Delta W = \frac{\alpha}{r}BA$6 expansion, the effective kernel update is

$\Delta W = \frac{\alpha}{r}BA$7

The deployed kernel is then simply $\Delta W = \frac{\alpha}{r}BA$8. Consequently, Conv-LoRA inherits the same no-additional-inference-latency property that distinguishes LoRA from sequential adapters: once merged, inference uses an ordinary convolution with an updated kernel.

3. Rank allocation, layer targeting, and optimization practice

In the original LoRA experiments on Transformers, a consistent finding was that adaptation should be distributed across multiple important weights with small rank rather than concentrated in a single weight with large rank. Under a fixed parameter budget of $\Delta W = \frac{\alpha}{r}BA$9M LoRA parameters on GPT-3 175B, adapting $$W_{0,\mathrm{mat}} \in \mathbb{R}^{C_{\text{out}} \times (C_{\text{in}}k_hk_w)}$$0 and $$W_{0,\mathrm{mat}} \in \mathbb{R}^{C_{\text{out}} \times (C_{\text{in}}k_hk_w)}$$1 with rank $$W_{0,\mathrm{mat}} \in \mathbb{R}^{C_{\text{out}} \times (C_{\text{in}}k_hk_w)}$$2 outperformed putting the same budget into a single projection with rank $$W_{0,\mathrm{mat}} \in \mathbb{R}^{C_{\text{out}} \times (C_{\text{in}}k_hk_w)}$$3, and adapting all four attention projections with rank $$W_{0,\mathrm{mat}} \in \mathbb{R}^{C_{\text{out}} \times (C_{\text{in}}k_hk_w)}$$4 was roughly similar in performance. The extracted design guidance for convolutional adaptation is therefore not to LoRA every convolution indiscriminately, but to start with the most “semantic” and bottleneck-like weights, such as $$W_{0,\mathrm{mat}} \in \mathbb{R}^{C_{\text{out}} \times (C_{\text{in}}k_hk_w)}$$5 pointwise convolutions, projection convolutions in bottleneck blocks, and later layers close to the representation head (Hu et al., 2021).

The same source also motivates small-rank regimes. In GPT-3 175B, ranks $$W_{0,\mathrm{mat}} \in \mathbb{R}^{C_{\text{out}} \times (C_{\text{in}}k_hk_w)}$$6 showed that even $$W_{0,\mathrm{mat}} \in \mathbb{R}^{C_{\text{out}} \times (C_{\text{in}}k_hk_w)}$$7 could be competitive for certain attention weights, while increasing rank beyond small values yielded diminishing returns. The extracted Conv-LoRA guidance is correspondingly to begin with small ranks, such as $$W_{0,\mathrm{mat}} \in \mathbb{R}^{C_{\text{out}} \times (C_{\text{in}}k_hk_w)}$$8 for big CNNs and $$W_{0,\mathrm{mat}} \in \mathbb{R}^{C_{\text{out}} \times (C_{\text{in}}k_hk_w)}$$9 for medium CNNs, and to increase rank only when the target task is sufficiently distinct from pretraining or when underfitting persists.

Optimization practice likewise transfers directly. The backbone kernel $W_0 \in \mathbb{R}^{d \times k}$0 is frozen; only the low-rank parameters are trained, along with any task-specific head. The original LoRA paper used AdamW and linear decay with warmup, and noted that LoRA typically uses larger learning rates than full fine-tuning because the optimized parameter set is small and stable. In the convolutional setting, this does not introduce a new objective: the model remains trained under the usual supervised or self-supervised loss, but optimization is restricted to the low-rank branch.

A common misconception is that Conv-LoRA requires a special convolution-specific theory of loss design. It does not. What changes is the parametrization of $W_0 \in \mathbb{R}^{d \times k}$1; the task loss, training data, and deployment semantics remain those of ordinary fine-tuning.

4. Spectral geometry and the limitations of standard Conv-LoRA

The SDS-LoRA analysis studies a single linear layer with

$W_0 \in \mathbb{R}^{d \times k}$2

where $W_0 \in \mathbb{R}^{d \times k}$3, $W_0 \in \mathbb{R}^{d \times k}$4, and $W_0 \in \mathbb{R}^{d \times k}$5 denotes the full fine-tuning gradient. The chain rule gives

$W_0 \in \mathbb{R}^{d \times k}$6

Writing rank-$W_0 \in \mathbb{R}^{d \times k}$7 SVDs

$W_0 \in \mathbb{R}^{d \times k}$8

yields

$W_0 \in \mathbb{R}^{d \times k}$9

From the full-weight viewpoint, the effective gradient induced by an SGD step is

$\Delta W \in \mathbb{R}^{d \times k}$0

The key conclusion is that the singular values appear squared in the effective gradient, so some directions are amplified while others are suppressed. The paper terms this anisotropic gradient scaling and argues that it distorts the full fine-tuning gradient by skewing it toward dominant singular directions while suppressing others (Oh et al., 15 Jun 2026).

This analysis is entirely matrix-based and therefore transfers directly to convolution after flattening the kernel to $\Delta W \in \mathbb{R}^{d \times k}$1. For Conv-LoRA, the same formulas apply to the matrixified kernel. The paper further defines effective rank by the stable rank

$\Delta W \in \mathbb{R}^{d \times k}$2

and reports that the effective rank of LoRA gradients is significantly lower once singular values act. This suggests that a standard Conv-LoRA module may only effectively use a few directions inside the nominal rank-$\Delta W \in \mathbb{R}^{d \times k}$3 subspace.

The geometric decomposition clarifies why. The orthonormal bases $\Delta W \in \mathbb{R}^{d \times k}$4 and the orthogonal factors $\Delta W \in \mathbb{R}^{d \times k}$5 are isometries; only $\Delta W \in \mathbb{R}^{d \times k}$6 introduce anisotropic scaling. Theorem 1 states that, with fixed row and column subspaces and fixed Frobenius norms $\Delta W \in \mathbb{R}^{d \times k}$7 and $\Delta W \in \mathbb{R}^{d \times k}$8, the gradient alignment $\Delta W \in \mathbb{R}^{d \times k}$9 is maximized when

$W_0 + \Delta W = W_0 + BA,$0

Under that condition, the effective gradient reduces to isotropic projection within the adapted subspaces. But the same source states that enforcing uniform singular values at all times is too restrictive because it forces the update itself to have uniform singular values, which severely limits representational capacity—especially problematic for structured kernels in conv layers. Standard Conv-LoRA is therefore efficient but spectrally imperfect: forward expressivity and backward geometry are coupled in a way that can degrade optimization.

5. SDS-LoRA and structurally decoupled convolutional adaptation

SDS-LoRA modifies the low-rank parameterization so that singular values remain available in the forward path but are structurally removed from the backward path. Its update is

$W_0 + \Delta W = W_0 + BA,$1

where $W_0 + \Delta W = W_0 + BA,$2 and $W_0 + \Delta W = W_0 + BA,$3 are orthonormal bases obtained from QR decompositions

$W_0 + \Delta W = W_0 + BA,$4

These bases are treated as constants during each backward pass and are periodically recomputed from the latest $W_0 + \Delta W = W_0 + BA,$5 and $W_0 + \Delta W = W_0 + BA,$6. The resulting gradients are

$W_0 + \Delta W = W_0 + BA,$7

Unlike standard LoRA, these expressions do not contain $W_0 + \Delta W = W_0 + BA,$8 or $W_0 + \Delta W = W_0 + BA,$9. The effective gradient becomes

$$B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k}, \qquad r \ll \min(d,k).$$0

where $$B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k}, \qquad r \ll \min(d,k).$$1 is a small error term induced by the slow evolution of $$B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k}, \qquad r \ll \min(d,k).$$2. Empirically, the cosine similarity between $$B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k}, \qquad r \ll \min(d,k).$$3 before and after updates remains $$B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k}, \qquad r \ll \min(d,k).$$4, so $$B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k}, \qquad r \ll \min(d,k).$$5 is reported as negligible (Oh et al., 15 Jun 2026).

For Conv-LoRA, the transfer is direct. A convolutional kernel is flattened, low-rank factors $$B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k}, \qquad r \ll \min(d,k).$$6 and $$B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k}, \qquad r \ll \min(d,k).$$7 are maintained in the flattened space, orthonormal bases $$B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k}, \qquad r \ll \min(d,k).$$8 are updated by QR, and the resulting $$B \in \mathbb{R}^{d \times r}, \qquad A \in \mathbb{R}^{r \times k}, \qquad r \ll \min(d,k).$$9 is reshaped back to kernel form. The extracted training algorithm is layer-local: warm up with standard LoRA for $A$0 iterations, compute a rank-$A$1 SVD of $A$2, reinitialize via

$A$3

reset optimizer states, then continue with SDS-LoRA while periodically updating the bases according to the progressively increasing interval schedule

$A$4

with $A$5.

The convergence theorem further distinguishes the two parameterizations. Under a $A$6-smooth loss, a local Polyak–Łojasiewicz condition, and an alignment parameter $A$7, the loss satisfies

$A$8

Thus LoRA’s rate degrades with the condition number $A$9 of the low-rank factors, whereas SDS-LoRA’s rate is independent of $W_0 + \Delta W$00. In convolutional terms, this indicates that a Conv-LoRA design which removes singular-value effects from the backward path can eliminate a conditioning penalty while retaining full rank-$W_0 + \Delta W$01 expressivity in the forward path.

6. Empirical evidence, misconceptions, and open questions

The direct empirical evidence for the anisotropic-scaling analysis comes from natural language and vision benchmarks in SDS-LoRA rather than from explicit CNN experiments. In the vision setting, the paper evaluates ViT-Base and ViT-Large, applying adapters to transformer modules such as query and value projections rather than to patch embeddings or conv stems. Nevertheless, the paper emphasizes that the analysis is layer-agnostic for any operator that is linear in its parameters, and therefore applies equally after convolutional kernel flattening. On ViT-Base with rank $W_0 + \Delta W$02, the reported averages are LoRA avg $W_0 + \Delta W$03, best LoRA variant $W_0 + \Delta W$04, SDS-LoRA $W_0 + \Delta W$05, and full fine-tuning $W_0 + \Delta W$06; at rank $W_0 + \Delta W$07, the best LoRA variant is $W_0 + \Delta W$08 and SDS-LoRA reaches $W_0 + \Delta W$09. ViT-Large shows the same qualitative pattern, with SDS-LoRA dominating LoRA variants and narrowing the gap to full fine-tuning (Oh et al., 15 Jun 2026).

These results establish several points relevant to Conv-LoRA. First, the anisotropic-scaling issue is not confined to LLMs; it is present in vision adaptation. Second, decoupling singular values from the backward path improves both convergence and final accuracy. Third, the absence of explicit CNN experiments should not be read as evidence against convolutional applicability. The paper states that nothing in the theory or implementation is tied to attention specifically.

Several misconceptions can therefore be addressed precisely. Conv-LoRA is not merely a heuristic “extra branch”; it is an additive low-rank parameterization of the kernel update that can be merged exactly into the base convolution. Orthonormal bases in SDS-LoRA do not imply that the forward update is constrained to uniform singular values; rather, the bases govern backward geometry while the unconstrained matrices $W_0 + \Delta W$10 and $W_0 + \Delta W$11 retain scale information for forward expressivity. Conversely, directly enforcing uniform singular values during training is not presented as the solution, because the same source states that such a constraint limits representational capacity.

Open issues remain. There is no explicit study of how spatial structure interacts with low-rank approximations of convolutional kernels. The extracted discussion identifies at least two possibilities: factoring along channels only, or factoring over the flattened space–channel dimension used in the paper. SDS-LoRA’s theory supports either choice, but the best unfolding may depend on the model. The convergence proof also assumes that the subspace-change term $W_0 + \Delta W$12 is negligible; this holds empirically in transformers, but for CNNs with very few adaptation steps or highly structured kernels, the required QR update frequency may need verification. These are not contradictions of the Conv-LoRA framework; they delimit the current evidence base and identify the main technical questions for further work.