Conditional Low-Rank Adaptation (CoLA)

• Conditional Low-Rank Adaptation (CoLA) is a set of techniques for low-rank network compression that conditions adaptation on contextual data or frozen weights.
• It employs context-aware SVD and TSQR methods to derive low-rank approximations that retain calibration data performance while reducing computational costs.
• The CondLoRA branch uses shared linear projections to generate task-adaptive low-rank updates, achieving significant parameter savings with minimal performance loss.

Conditional Low-Rank Adaptation (CoLA) encompasses a class of low-rank adaptation techniques for neural networks in which the adaptation or compression is conditioned on either contextual data (e.g., input activations) or frozen network parameters. "Conditional" refers to either context-aware low-rank approximation schemes for data-dependent model compression or Conditionally Parameterized LoRA, which generates low-rank updates from the original weight matrices. Both branches share the goal of reducing parameter count or improving fine-tuning efficiency while maintaining or enhancing target-specific accuracy.

1. Formal Problem Statement

Conditional Low-Rank Adaptation techniques target the following scenario: given a pretrained weight matrix $W \in \mathbb{R}^{m \times n}$ (typically a layer in a transformer or other neural network) and a "calibration" or context matrix $X \in \mathbb{R}^{n \times k}$ (which could be a set of input activations or a function of the original weights), find a low-rank matrix $W'$ of rank at most $r$ such that the discrepancy over the calibration data is minimized: $\min_{\mathrm{rank}(W') \leq r} \| WX - W'X \|_F^2$ This can be equivalently formulated as a weighted low-rank approximation: $$\|W - W'\|_X^2 = \mathrm{Tr}\big( (W - W')^T XX^T (W - W') \big)$$ Unlike standard SVD-based compression that minimizes $\|W - W'\|_F^2$, the CoLA objective preserves performance specifically on the calibration data by conditioning the compression on the input distribution (Parkina et al., 10 Jul 2025).

A distinct but related aim is addressed in Conditionally Parameterized LoRA: to generate task-adaptive low-rank matrices ($A, B$) conditioned on the original network weights $W_0$ via a single learnable linear mapping, significantly reducing parameter overhead while matching standard LoRA performance (Kim et al., 2024).

2. Context-Aware Low-Rank Approximation Methods

Conventional context-aware low-rank approximation (CoLA) methods form the Gram matrix $G = X X^T$ and perform SVD or Cholesky decompositions to construct the low-rank projection. For example,

$X \in \mathbb{R}^{n \times k}$0

This strategy suffers from two principal limitations:

• Gram formation squares the condition number of $X \in \mathbb{R}^{n \times k}$1, causing loss of numerical precision or overt singularities, especially when $X \in \mathbb{R}^{n \times k}$2 is nearly singular or high-dimensional.
• Computational cost can be prohibitive due to both the $X \in \mathbb{R}^{n \times k}$3 time and $X \in \mathbb{R}^{n \times k}$4 memory complexity of forming and inverting $X \in \mathbb{R}^{n \times k}$5.

The COALA framework introduces an inversion-free and regularized approach. The optimal low-rank adaptation bypasses Gram/inverse computation: $X \in \mathbb{R}^{n \times k}$6 For efficiency, when $X \in \mathbb{R}^{n \times k}$7, a tall-skinny QR (TSQR) is performed on $X \in \mathbb{R}^{n \times k}$8, yielding $X \in \mathbb{R}^{n \times k}$9, and SVD is executed on $W'$0 to retrieve the top singular vectors (Parkina et al., 10 Jul 2025). Regularization with a Tikhonov term,

$W'$1

is equivalent to unregularized CoLA on the augmented calibration matrix $W'$2.

3. Conditionally Parameterized LoRA (CondLoRA)

Conditional Low-Rank Adaptation also encompasses the CondLoRA model, where task-adaptive low-rank matrices are generated from a (frozen) pretrained matrix $W'$3 by shared linear projections: $W'$4 where $W'$5 are learned matrices shared across all layers of a given module type (e.g., "query," "value"). The low-rank adaptation at each layer $W'$6 for module type $W'$7 is then

$W'$8

This design is motivated by empirical findings that the conversion mappings $W'$9 in standard LoRA are highly similar across layers. Instead of independently learning $r$0 for every layer, CondLoRA parameterizes all low-rank updates using a single linear map per module type, yielding significant parameter savings—approximately 12-fold in standard transformer architectures—without statistically significant loss in downstream performance (Kim et al., 2024).

4. Algorithmic Procedures

The key algorithms follow the regime:

Context-Aware Low-Rank Approximation (COALA):

• Input: Weight matrix $r$1, calibration matrix $r$2, target rank $r$3, regularization parameter $r$4.
• TSQR computes $r$5.
• SVD on $r$6 yields the top $r$7 singular vectors $r$8.
• The optimal low-rank weight is $r$9.
• For regularized adaptation, use the augmented calibration matrix $\min_{\mathrm{rank}(W') \leq r} \| WX - W'X \|_F^2$0.

Conditional LoRA (CondLoRA):

• For each module, learn $\min_{\mathrm{rank}(W') \leq r} \| WX - W'X \|_F^2$1.
• At each layer, compute $\min_{\mathrm{rank}(W') \leq r} \| WX - W'X \|_F^2$2 and $\min_{\mathrm{rank}(W') \leq r} \| WX - W'X \|_F^2$3 via linear projections of $\min_{\mathrm{rank}(W') \leq r} \| WX - W'X \|_F^2$4.
• The fine-tuned weight is $\min_{\mathrm{rank}(W') \leq r} \| WX - W'X \|_F^2$5.

Pseudocode Snapshots

Method	Key Steps Summary
COALA	TSQR on $\min_{\mathrm{rank}(W') \leq r} \| WX - W'X \|_F^2$6; SVD on $\min_{\mathrm{rank}(W') \leq r} \| WX - W'X \|_F^2$7; construct $\min_{\mathrm{rank}(W') \leq r} \| WX - W'X \|_F^2$8
Regularized COALA	Form $\min_{\mathrm{rank}(W') \leq r} \| WX - W'X \|_F^2$9; use COALA on $$\|W - W'\|_X^2 = \mathrm{Tr}\big( (W - W')^T XX^T (W - W') \big)$$0
CondLoRA	Compute $$\|W - W'\|_X^2 = \mathrm{Tr}\big( (W - W')^T XX^T (W - W') \big)$$1, $$\|W - W'\|_X^2 = \mathrm{Tr}\big( (W - W')^T XX^T (W - W') \big)$$2; set $$\|W - W'\|_X^2 = \mathrm{Tr}\big( (W - W')^T XX^T (W - W') \big)$$3

5. Theoretical Guarantees

The COALA framework provides explicit error bounds ensuring robust convergence to the unregularized solution as $$\|W - W'\|_X^2 = \mathrm{Tr}\big( (W - W')^T XX^T (W - W') \big)$$4, even in the presence of highly rank-deficient or nearly singular $$\|W - W'\|_X^2 = \mathrm{Tr}\big( (W - W')^T XX^T (W - W') \big)$$5. For instance, letting $$\|W - W'\|_X^2 = \mathrm{Tr}\big( (W - W')^T XX^T (W - W') \big)$$6: $$\|W - W'\|_X^2 = \mathrm{Tr}\big( (W - W')^T XX^T (W - W') \big)$$7 A more general bound in the rank-deficient case maintains linear convergence in $$\|W - W'\|_X^2 = \mathrm{Tr}\big( (W - W')^T XX^T (W - W') \big)$$8 with explicit conditioning dependence. These results ensure stability even for extremely tall and ill-conditioned calibration matrices (Parkina et al., 10 Jul 2025).

For CondLoRA, the theoretical justification is empirical: normalized subspace similarity ($$\|W - W'\|_X^2 = \mathrm{Tr}\big( (W - W')^T XX^T (W - W') \big)$$9) among per-layer conversion matrices demonstrates that a single pair of projection matrices per module type can generate effective low-rank updates, realizing significant parameter efficiency without loss of adaptation quality (Kim et al., 2024).

6. Empirical Performance and Efficiency

Empirical evaluations establish that COALA is both more numerically stable and computationally efficient than Gram-inverse-based SVD methods. For LLaMA3-1B with 64 calibration samples, COALA executes in approximately 196 s versus 274 s for SVD-LLM, while for LLaMA3-8B (128 samples) the speeds are 1,811 s (COALA) versus 3,625 s (SVD-LLM). Relative to earlier methods, COALA consistently achieves lower approximation error, especially at low rank or with ill-conditioned data.

Compression to 70% size on LLaMA3-8B using regularized COALA ($\|W - W'\|_F^2$0) yields accuracy improvements on reasoning benchmarks—e.g., +3.0% (PIQA), +2.7% (ARC-E)—over ASVD, SVD-LLM, and unregularized COALA. Similar improvements are observed on Mistral-7B models (Parkina et al., 10 Jul 2025).

On the task-adaptation front, CondLoRA achieves a GLUE benchmark average of 83.42, versus 83.38 for full-parameter LoRA, using only $\|W - W'\|_F^2$11/12 the trainable parameters and with minor gains in training speed (Kim et al., 2024). Task-wise scores differ by at most ±1 point, with differences not statistically significant ($\|W - W'\|_F^2$2).

7. Connections, Extensions, and Significance

Conditional Low-Rank Adaptation unites principled approaches for data-aware and weight-aware compression, sharing a core insight: adaptation matrices can—in both input- and weight-conditioned settings—be expressed by linear maps that respect the intrinsic geometry of the pretrained model or the relevant data subspace. This paradigm covers both numerically robust model compression (COALA) and parameter-efficient fine-tuning regimes where low-rank matrices are generated conditionally via global linear projectors (CondLoRA).

The COALA framework also generalizes regularized and data-scarce settings, achieving superior unpredictability in real-world deployment scenarios (including large-scale, memory-bound calibration or severe data scarcity).

This suggests that CoLA methodologies provide a unified and robust foundation for both context-driven model adaptation and parameter-efficient transfer in modern large-scale neural networks. Further, as challenges of efficient, robust, and scalable adaptation continue to intensify, the conditional approaches outlined offer a canonical toolkit for both empirical and theoretical advancements in PEFT, model compression, and initialization for lightweight fine-tuning (Parkina et al., 10 Jul 2025, Kim et al., 2024).