LoRA-Squeeze: Efficient Rank Transformation

• LoRA-Squeeze is a method that learns a high-rank LoRA adapter and then compresses it using RSVD to overcome the challenges of preset rank selection.
• It offers variants like Post-Squeeze, Cont-Squeeze, and Memory-efficient Post-Squeeze, employing techniques such as QR decomposition for resource-efficient compression.
• Empirical results across text and vision-language tasks demonstrate that high-rank training followed by gradual compression can match or surpass direct low-rank training performance.

LoRA-Squeeze is a methodology for changing the rank of standard LoRA adapters either post-hoc after fine-tuning or dynamically during training. It fine-tunes LoRA with a deliberately higher source rank, reconstructs or efficiently approximates the full weight update matrix, and then uses Randomized Singular Value Decomposition (RSVD) to create a new compressed LoRA module at a lower target rank. The method is motivated by the proposition that it is better to first learn an expressive, higher-rank solution and then compress it, rather than learning a constrained, low-rank solution directly (Vulić et al., 11 Feb 2026).

1. Standard LoRA setting and the rank-selection problem

In the formulation used for LoRA-Squeeze, a pretrained weight matrix $W_0 \in \mathbb{R}^{m \times n}$ is adapted by a low-rank update $A B$, where $A \in \mathbb{R}^{m \times r}$, $B \in \mathbb{R}^{r \times n}$, and $r \ll \min(m,n)$. The adapted weight is written as $W_T \approx W_0 + A B$, and in transformer layers the corresponding linear map becomes $y = (W_0 + A B)x$, with $W_0$ frozen and only $A,B$ trainable (Vulić et al., 11 Feb 2026).

The practical problem addressed by LoRA-Squeeze is not the validity of low-rank adaptation itself, but the difficulty of choosing the rank $r$ before training. The reported motivation emphasizes four points: pre-selecting a good rank is difficult, optimal hyperparameters depend on rank, heterogeneous ranks complicate deployment, and direct training at very low rank is a highly constrained non-convex optimization problem. The proposed response is to decouple training rank from deployment rank by first learning with a higher source rank $A B$0 and only later compressing to a lower target rank $A B$1 (Vulić et al., 11 Feb 2026).

The experimental setting in the LoRA-Squeeze study is deliberately conservative with respect to architecture changes. For all experiments, LoRA is applied only to text components of Gemma 3 models. In the vision-language setting, the SigLIP vision encoder and the multimodal projection layers remain frozen; only the text layers receive LoRA (Vulić et al., 11 Feb 2026).

2. Post-hoc compression by RSVD

The post-hoc variant, termed Post-Squeeze, begins from a LoRA adapter trained at source rank $A B$2. For each adapted weight, the learned update is first reconstructed in full form as

$A B$3

RSVD is then applied with target rank $A B$4, yielding an approximate factorization

$A B$5

The new target-rank LoRA factors are obtained by symmetrically splitting the singular values: $A B$6 so that $A B$7 (Vulić et al., 11 Feb 2026).

A memory-efficient variant avoids explicitly constructing $A B$8. It QR-decomposes the source factors as

$A B$9

and rewrites the update as

$A \in \mathbb{R}^{m \times r}$0

SVD or RSVD is then performed on the small matrix $A \in \mathbb{R}^{m \times r}$1, after which the target-rank factors are reconstructed as

$A \in \mathbb{R}^{m \times r}$2

The study states that this is mathematically equivalent to SVD on $A \in \mathbb{R}^{m \times r}$3 but scales in $A \in \mathbb{R}^{m \times r}$4 rather than $A \in \mathbb{R}^{m \times r}$5 (Vulić et al., 11 Feb 2026).

Three variants are distinguished in practice.

Variant	Procedure	Purpose
Post-Squeeze	Compress $A \in \mathbb{R}^{m \times r}$6 after training	Post-hoc rank reduction
Cont-Squeeze	Post-Squeeze plus continued fine-tuning	Recover aggressive compression
Memory-efficient Post-Squeeze	Compress via QR/SVD in $A \in \mathbb{R}^{m \times r}$7 space	Lower memory and compute

RSVD is used because truncated SVD is the optimal low-rank approximation of a matrix, while exact SVD on large $A \in \mathbb{R}^{m \times r}$8 weights is expensive. The reported RSVD hyperparameters are oversampling $A \in \mathbb{R}^{m \times r}$9 and power iterations $B \in \mathbb{R}^{r \times n}$0, and the study reports negligible differences between RSVD and full SVD in final performance (Vulić et al., 11 Feb 2026).

3. In-tuning rank annealing

LoRA-Squeeze also defines an in-tuning variant, In-Squeeze, in which compression is performed repeatedly during fine-tuning rather than once at the end. Training starts at a high rank such as $B \in \mathbb{R}^{r \times n}$1, then compresses along a schedule such as

$B \in \mathbb{R}^{r \times n}$2

After each compression step, training continues at the new lower rank (Vulić et al., 11 Feb 2026).

Two scheduling rules are reported. In the Standard scheme, the number of training steps allocated to a rank $B \in \mathbb{R}^{r \times n}$3 is proportional to $B \in \mathbb{R}^{r \times n}$4. In the Minimum Steps scheme, each stage receives at least 200 steps, and the remaining steps are distributed proportionally to rank. The Minimum Steps scheme is reported to work best, with the interpretation that lower ranks need enough time to adapt after each squeeze (Vulić et al., 11 Feb 2026).

Cont-Squeeze is the special case of a single compression followed by a short continuation phase. This is particularly important for severe transformations such as $B \in \mathbb{R}^{r \times n}$5, where one-shot compression can initially collapse performance. The reported explanation is that each squeeze discards singular directions with small singular values, and smaller incremental drops lose less information at a time; subsequent fine-tuning can then compensate within the new rank budget (Vulić et al., 11 Feb 2026).

The paper also introduces a retained-variance diagnostic,

$B \in \mathbb{R}^{r \times n}$6

and states that collapse correlates with low retained variance. This suggests that singular-value energy can function as a proxy for how aggressive a squeeze step can be before the transformation gap becomes too large (Vulić et al., 11 Feb 2026).

4. Computational characteristics and deployment behavior

The computational comparison reported for a representative case with $B \in \mathbb{R}^{r \times n}$7, $B \in \mathbb{R}^{r \times n}$8, $B \in \mathbb{R}^{r \times n}$9, and $r \ll \min(m,n)$0 highlights the difference between exact and approximate compression. Full SVD is reported as $r \ll \min(m,n)$1B FLOPs, RSVD on $r \ll \min(m,n)$2 as $r \ll \min(m,n)$3M FLOPs, and memory-efficient RSVD as $r \ll \min(m,n)$4M FLOPs. The paper therefore characterizes the memory-efficient form as both memory and compute friendly (Vulić et al., 11 Feb 2026).

After squeezing, the deployed module remains a standard LoRA at rank $r \ll \min(m,n)$5. Inference behavior is unchanged; only $r \ll \min(m,n)$6 differ. No special runtime support is needed beyond standard LoRA support. This is an important distinguishing property, because many adaptive-rank or shared-parameter PEFT methods alter either module structure or per-layer heterogeneity at deployment time (Vulić et al., 11 Feb 2026).

The method is also framed as a way to restore homogeneous deployment ranks. During training, different LoRAs may have different source ranks, but Post-Squeeze or In-Squeeze can standardize them to a single target rank. The paper explicitly links this to serving simplicity for systems such as SLoRA and Punica, where heterogeneous ranks and exotic LoRA variants complicate batching and I/O (Vulić et al., 11 Feb 2026).

Practical training details further underscore the method’s role as a drop-in extension to standard LoRA. The reported optimizer is Adafactor with a 1000-step linear warmup and no learning-rate decay. The learning-rate grid is $r \ll \min(m,n)$7, and the study reports that lower ranks benefit from higher learning rates, with $r \ll \min(m,n)$8 favoring $r \ll \min(m,n)$9 and higher ranks favoring $W_T \approx W_0 + A B$0. The training budget is 10k steps for smaller datasets and 15k for larger ones, with batch size 8 and sequence length 1024 (Vulić et al., 11 Feb 2026).

5. Experimental evidence across text and vision-language tasks

The empirical study spans 13 text tasks—WinoGrande-Large, BoolQ, PIQA, DROP, ANLI-r2, PAWS Wiki, HellaSwag, OpenBookQA, GoEmotions, ARC-Easy, ARC-Challenge, SocialIQA, and MMLU—and 10 vision-language tasks—AI2D, A-OKVQA, CountBenchQA, DocVQA, InfographicVQA, OCR-VQA, OK-VQA, ScienceQA, TallyQA, and TextVQA. The main model is Gemma 3 4B IT, with additional experiments on Gemma 3 1B IT and limited validation on Gemma 3 12B IT (Vulić et al., 11 Feb 2026).

A central comparison is against direct LoRA trained at the target rank with per-rank optimized learning rates. For moderate compression pairs, Post-Squeeze frequently matches or exceeds direct low-rank training. On Gemma 4B IT over text tasks, at target rank 4 the average Direct(4) score is 90.13, whereas Post-Squeeze(16→4) reaches 90.41; at target rank 8 the average Direct(8) score is 90.30, whereas Post-Squeeze(128→8) reaches 90.63 (Vulić et al., 11 Feb 2026).

The most detailed rank-1 result illustrates the distinction between one-shot compression and annealed compression. On Gemma 4B IT over text tasks, Direct LoRA at $W_T \approx W_0 + A B$1 gives 90.08 with no extra steps, 89.74 with +200 steps, and 89.67 with +700 steps. Post-Squeeze $W_T \approx W_0 + A B$2 gives 77.32 at +0 steps, but recovers to 90.05 with +200 steps and 90.09 with +700 steps. In-Squeeze reaches 90.21 with the Standard schedule and 90.51 with the Minimum Steps schedule, which is the best overall average in that comparison (Vulić et al., 11 Feb 2026).

The same pattern is reported beyond the main model. On the 1B model, In-Squeeze again gives the best average, 83.80 versus 83.39 for direct training. On the vision-language tasks, gains are smaller but persistent, with average direct performance at 85.20 and In-Squeeze at approximately 85.48–85.51. The study also reports standard errors over three random seeds—for example, 90.21 ± 0.061 at rank 1 and 90.55 ± 0.075 at rank 32—arguing that the gains attributed to LoRA-Squeeze exceed seed variance (Vulić et al., 11 Feb 2026).

A separate practical scenario concerns hyperparameter search. When the learning rate is tuned only once at a high source rank and then reused for direct low-rank training, direct low-rank LoRA is reported to underperform substantially. In that setting, Post-Squeeze from the high-rank model consistently outperforms direct low-rank runs even with zero extra training at the target rank. The paper’s stated implication is that one can tune hyperparameters once at a high rank and derive a family of low-rank adapters by squeezing (Vulić et al., 11 Feb 2026).

6. Relation to adjacent compression directions

LoRA-Squeeze belongs to a broader family of efforts that reduce LoRA overhead, but it operates on a different axis from most earlier methods. LoRA-drop evaluates importance via activations $W_T \approx W_0 + A B$3, retains independent LoRA in important layers, and replaces low-importance layers with shared adapters; it is therefore a layer-selection and sharing method rather than a post-hoc rank-transformation method (Zhou et al., 2024). S2-LoRA, or Sparsely Shared LoRA, shares low-rank bases across modules and moves most task-specific capacity into sparse diagonal rank coefficients; it compresses by basis sharing and sparsity rather than by learning at high rank and compressing afterward (Liu et al., 2023).

A different line of work is metric-driven rather than algorithmic. The MIUB scaling-law study proposes a Mutual Information Upper Bound between hidden spaces of the frozen model and LoRA, and argues that lower MIUB tracks better adaptation as model size, LoRA rank, and data size increase. That work does not define a compression operator, but it provides a principled internal metric for deciding how much LoRA is needed (Zhang et al., 6 Jan 2025). LowRA, by contrast, targets the base model rather than the adapter: it enables LoRA fine-tuning below 2 bits per parameter by mixed 1/2/4-bit quantization with learned thresholds, mappings, hierarchical ILP precision assignment, and CUDA kernels (Zhou et al., 12 Feb 2025).

Other neighboring directions address different deployment constraints. Compression-aware LoRA on compressed backbones, instantiated as CPET, adapts LoRA to a compressed model through knowledge inheritance, recovery modules, and distillation; the goal there is to recover performance lost by backbone compression rather than to change adapter rank (Zhao et al., 2023). Semantic-guided LoRA Parameter Generation uses task descriptions and a conditional VAE to generate task-specific LoRA parameters without additional training on user tasks; its compression effect comes from replacing many stored adapters with a semantic generator and a small expert bank (Li et al., 5 Sep 2025).

The limitations reported for LoRA-Squeeze are correspondingly specific. One-step aggressive compression such as $W_T \approx W_0 + A B$4 can produce performance collapse. Minimum-steps annealing consistently performs better than the proportional schedule, indicating schedule sensitivity. The method is developed and evaluated for standard LoRA; extending it to more complex LoRA derivatives is left for future research (Vulić et al., 11 Feb 2026). A common misconception is therefore to treat LoRA-Squeeze as a generalized importance allocator or as a new inference architecture. In the reported formulation it is neither: it is a rank-changing procedure for standard LoRA, and the final deployed module remains standard LoRA at the target rank (Vulić et al., 11 Feb 2026).