LIFT: Low-Rank Informed Sparse Fine-Tuning

• The paper introduces LIFT, a method that employs low-rank approximation to identify critical 'Principal Weights' for sparse fine-tuning, updating only 5% of model parameters.
• By dynamically refreshing a sparse mask and maintaining only selected weight updates, LIFT enhances reasoning performance while preserving memory efficiency.
• Experimental results demonstrate that LIFT outperforms Full FT and LoRA on reasoning, language understanding, and code generation tasks, retaining up to 20% more source-domain knowledge.

Low-Rank Informed Sparse Fine-Tuning (LIFT) is a sparse fine-tuning method for LLMs designed for reasoning-focused supervised fine-tuning. Its defining procedure is to low-rank approximate each trainable weight matrix, identify the largest-magnitude entries in that approximation as “Principal Weights,” and fine-tune only those corresponding weights in the original model. The central claim is that magnitude-based sparse fine-tuning is weak on raw weights but becomes highly effective after rank reduction. In the reported experiments, LIFT updates only the top 5% Principal Weights throughout training, achieves better performance on reasoning tasks than Full FT, maintains memory efficiency on par with popular parameter-efficient fine-tuning methods, and retains up to 20% more source-domain knowledge compared to Full FT and LoRA (Liu et al., 1 Jun 2025).

1. Definition and problem setting

LIFT addresses the trade-off between adaptation strength and efficiency in reasoning-focused supervised fine-tuning. Full FT updates all parameters, has the highest memory cost, and is described as being susceptible to overfitting and catastrophic forgetting, particularly when data is limited. Sparse fine-tuning offers a more efficient alternative, but the paper argues that earlier sparse methods have lagged in the LLM setting because it is difficult to identify parameters that are truly critical for reasoning (Liu et al., 1 Jun 2025).

The method is built around a specific hypothesis: weights with the largest magnitude after low-rank approximation are critical weights for fine-tuning, termed Principal Weights. This distinguishes LIFT from generic sparse fine-tuning heuristics such as raw magnitude, movement score, gradient magnitude, or random masking. The paper states that plain magnitude-based sparse FT is weak, but magnitude after low-rank approximation becomes powerful (Liu et al., 1 Jun 2025).

In this formulation, LIFT is sparse but not low-rank in the same sense as LoRA. It uses low-rank structure to select important original weights, then directly updates those original weights. This suggests that low-rank approximation serves as a denoising or structure-revealing step rather than as the final parameterization of the update.

2. Low-rank approximation and Principal Weights

For each trainable matrix $W_i$, LIFT computes a rank-$r$ approximation

$$W_i' = \arg\min_{\operatorname{rank}(W_i') \le r}\|W_i - W_i'\|_F$$

using SVD and retaining the top $r$ singular values and vectors. The paper presents this as the standard best rank-$r$ approximation in Frobenius norm, justified by the Eckart–Young–Mirsky theorem (Liu et al., 1 Jun 2025).

The method’s interpretation is that top singular directions or lower-order components contain task-relevant and contextual knowledge, whereas higher-order components often act like noise. After rank reduction, the remaining matrix is treated as a cleaned-up representation of the model’s strongest encoded structure. The largest-magnitude entries in $W_i'$ are then interpreted as the most important coefficients of that structure, and these entries define the Principal Weights (Liu et al., 1 Jun 2025).

Given $W'$, LIFT constructs a binary mask $M$ by selecting the top-$k$ entries of $W'$ by absolute magnitude: $r$0 In the main experiments, the selected fraction is typically the top 5% of weights. The selection rule is therefore: rank-reduce $r$1, rank entries of $r$2 by magnitude, keep the top fraction, and use that mask on the original matrix $r$3 (Liu et al., 1 Jun 2025).

A central conceptual point is that LIFT does not update the low-rank matrix $r$4 itself. Instead, it uses $r$5 only to identify which entries of the original matrix $r$6 should remain trainable. This is the sense in which the method is “low-rank informed” rather than itself a low-rank adapter.

3. Sparse optimization and dynamic mask refresh

LIFT stores and updates only the masked components of each gradient. If $r$7 is the gradient of $r$8 at iteration $r$9, then the method keeps only

$$W_i' = \arg\min_{\operatorname{rank}(W_i') \le r}\|W_i - W_i'\|_F$$0

According to the appendix summary, the training loop is an Adam-style sparse optimization procedure: initialize the mask with calc_mask(\theta_0), compute gradients at each step, keep only entries where $$W_i' = \arg\min_{\operatorname{rank}(W_i') \le r}\|W_i - W_i'\|_F$$1, update sparse momentum and variance states only for those entries, and periodically recompute the mask from the current parameters (Liu et al., 1 Jun 2025).

The mask is explicitly dynamic rather than fixed. It is refreshed every update_mask_interval steps because the low-rank approximation and the important entries can change during training. This periodic recomputation is part of the method rather than a minor implementation detail, since the definition of Principal Weights depends on the evolving parameter matrices (Liu et al., 1 Jun 2025).

The reported training procedure can be summarized in six stages: start from a pretrained LLM; compute a rank-$$W_i' = \arg\min_{\operatorname{rank}(W_i') \le r}\|W_i - W_i'\|_F$$2 approximation for each trainable matrix; select Principal Weights by top-$$W_i' = \arg\min_{\operatorname{rank}(W_i') \le r}\|W_i - W_i'\|_F$$3 magnitude in the approximated matrix; fine-tune only the corresponding weights in the original matrix; maintain Adam states only for selected weights; and periodically refresh the mask. Because only selected weights receive gradients and optimizer states, the method reduces optimizer-state memory substantially (Liu et al., 1 Jun 2025).

4. Empirical behavior on reasoning, language understanding, code, and retention

The paper reports that LIFT consistently matches or exceeds Full FT across several task families. On GPQA Diamond with Qwen2.5 reasoning models fine-tuned on s1K, Qwen2.5-1.5B improves from 26.77 with Full FT to 28.79 with LIFT, and Qwen2.5-3B improves from 33.33 to 34.85 (Liu et al., 1 Jun 2025).

On commonsense reasoning using Commonsense-170K, LIFT is reported as best overall on the listed LLaMA models. For LLaMA-2-7B, Full FT has average 83.53 and LIFT has 84.66. For LLaMA-3-8B, Full FT has average 86.64 and LIFT has 87.88. The paper states that LIFT improves over LoRA by up to 4.42% on commonsense reasoning and over Full FT by about 1.13–1.24% overall depending on model (Liu et al., 1 Jun 2025).

On arithmetic reasoning with MATH-10K, the method is described as consistently strongest or near-strongest. For LLaMA-3.2-3B, Full FT average is 76.45, LoRA average is 75.71, and LIFT average is 77.06. For LLaMA-3-8B, Full FT average is 80.18 and LIFT average is 81.78. The paper emphasizes especially large gains on harder tasks such as GSM8K and SVAMP (Liu et al., 1 Jun 2025).

On natural language understanding, LIFT is reported to achieve the best score on every listed GLUE task for DeBERTa-v3-base, with average 89.24 compared with 88.36 for Full FT. On code generation after Alpaca fine-tuning, the reported HumanEval metrics are Pass@1 16.46 for LIFT versus 15.24 for Full FT, and Pass@10 31.10 for LIFT versus 28.05 for Full FT. On StrategyQA, the reported scores are 72.53 versus 70.61 on LLaMA-2-7B and 75.85 versus 74.81 on LLaMA-3-8B (Liu et al., 1 Jun 2025).

A separate generalization study addresses source-domain retention. For arithmetic fine-tuning, LIFT outperforms Full FT and LoRA on target-domain tasks, while on the source domain it is reported to be more than 5% better than Full FT and about 12% better than LoRA in one setting; in a larger 8B setting it is reported as 10%+ better than Full FT and almost 20% better than LoRA. The abstract-level summary of this result is that LIFT retains up to 20% more source-domain knowledge than Full FT and LoRA (Liu et al., 1 Jun 2025).

On the efficiency side, the optimizer-state memory is reported to drop from about 27 GB in Full FT to 1.3 GB on LLaMA-2-7B, which is described as under 5% of Full FT. The paper characterizes this as memory efficiency roughly comparable to LoRA (Liu et al., 1 Jun 2025).

5. Mechanistic analyses and ablations

The paper provides several analyses intended to justify the Principal Weight hypothesis. In perturbation experiments, small Gaussian noise is added to different sets of parameters. When the noise is added to LIFT-selected parameters, Wikitext perplexity rises sharply, next-token prediction collapses much more than with other selection rules, and arithmetic reasoning accuracy drops drastically, even to near zero in the perturbation experiment. This is presented as evidence that the selected weights are unusually critical (Liu et al., 1 Jun 2025).

The method is also examined through eigenspace rotation. The reported result is that LIFT causes larger deviation in eigenspace than LoRA and Full FT in important layers, especially Output, Up projection, and Down projection. The paper interprets this as evidence that LIFT can meaningfully alter the model’s principal directions during adaptation (Liu et al., 1 Jun 2025).

A related analysis concerns update rank. The update matrix under LIFT is reported to have much larger rank than LoRA and rank close to Full FT in crucial layers. The appendix also argues that LIFT can significantly alter spectral norm more than other sparse selection methods, with the suggested implication that this matters for generalization and learning dynamics (Liu et al., 1 Jun 2025).

The ablations emphasize that the informative component is not sparsity alone, but the low-rank-informed choice of sparse coordinates. On GSM8K, LIFT is reported to outperform random selection, gradient magnitude, movement score, and plain weight magnitude. In a rank-reduction ablation, preserving the largest singular values/components works best, while random, smallest, or hybrid rank-preservation strategies perform worse (Liu et al., 1 Jun 2025).

These analyses support the paper’s broader explanation: low-rank approximation filters out noise, exposes informative structure, and turns magnitude into a more reliable importance proxy than it is on raw weights. This suggests that LIFT’s empirical gains come from selecting sparse coordinates that remain aligned with principal task-relevant structure.

6. Relation to LoRA, selective sparse PEFT, and acronym overlap

LIFT differs sharply from LoRA even though both involve low-rank structure. LoRA adds low-rank update matrices $$W_i' = \arg\min_{\operatorname{rank}(W_i') \le r}\|W_i - W_i'\|_F$$4, trains those adapters instead of original weights, and constrains the update through the low-rank form. LIFT does not constrain the update to be low-rank; it updates selected original weights chosen via low-rank-informed importance. The paper further reports that the resulting update matrix under LIFT has larger magnitude, higher rank, and greater ability to rotate eigenspaces than LoRA, while remaining sparse (Liu et al., 1 Jun 2025).

The method also differs from selective LoRA-style retention methods in vision and vision-LLMs. One such approach extends LoRA with a learned indicator function

$$W_i' = \arg\min_{\operatorname{rank}(W_i') \le r}\|W_i - W_i'\|_F$$5

and an $$W_i' = \arg\min_{\operatorname{rank}(W_i') \le r}\|W_i - W_i'\|_F$$6-style sparsity penalty

$$W_i' = \arg\min_{\operatorname{rank}(W_i') \le r}\|W_i - W_i'\|_F$$7

thereby activating only a small subset of LoRA blocks to reduce forgetting and preserve zero-shot and OOD behavior. That method also reports that effective fine-tuning can be achieved with as few as 5% of active blocks, but its sparsity acts at the level of adapter blocks rather than principal entries of original weight matrices (Bafghi et al., 26 Jan 2025).

RoseLoRA offers another adjacent design. It keeps the LoRA parameterization $$W_i' = \arg\min_{\operatorname{rank}(W_i') \le r}\|W_i - W_i'\|_F$$8 but imposes sparsity intended to survive into the product $$W_i' = \arg\min_{\operatorname{rank}(W_i') \le r}\|W_i - W_i'\|_F$$9, using row-wise sparsity on $r$0 and column-wise sparsity on $r$1. Its motivation is selective updating for knowledge editing and preserving unrelated knowledge. This is close in spirit to LIFT-style sparse low-rank adaptation, but the mechanism is product sparsification rather than Principal Weight selection in original matrices (Wang et al., 2024).

A further related method is SPruFT, a structured-pruning-based sparse fine-tuning framework that identifies important neurons or channels using pruning-inspired scores and fine-tunes only the weights associated with those selected units. Its structure is row-based rather than low-rank-informed. The paper describes this as conceptually close to LIFT-style ideas because both use importance-guided sparse adaptation, but SPruFT selects rows or neurons rather than magnitude-ranked entries after rank reduction (Li et al., 17 Feb 2025).

The acronym “LIFT” also has a separate meaning in diffusion LLMs: Learnability-Informed Fine-Tuning. That method selects supervised tokens according to diffusion time and model confidence, learning easy/common tokens when the input is heavily masked and hard/rare tokens when more context is available. Despite the identical acronym, it is a distinct method centered on token-level supervision schedules rather than sparse weight updates (Parashar et al., 21 May 2026).

7. Interpretation, limitations, and significance

The reported practical interpretation of LIFT is fourfold: low-rank approximation filters out noise and exposes important structure; the largest-magnitude entries in the denoised matrix are the Principal Weights; updating only these weights focuses training on informative directions; and this combination yields strong task adaptation, better retention of pretrained knowledge than Full FT, and lower memory usage (Liu et al., 1 Jun 2025).

The paper’s broader significance lies in its challenge to two common assumptions. First, it argues that sparse fine-tuning for LLMs need not rely on raw heuristics such as weight magnitude or movement score. Second, it suggests that low-rank structure is useful not only as an adapter parameterization, as in LoRA, but also as an importance estimator for sparse direct updates. A plausible implication is that low-rank approximation can function as a structure extractor even when the final update is neither dense nor low-rank.

The method’s scope is nevertheless specific. It is presented for reasoning-focused supervised fine-tuning, and its main empirical claims are concentrated on reasoning benchmarks, GLUE, code generation, and source-domain retention. The paper also notes an efficiency trade-off in mask refresh frequency: updating the mask too frequently hurts efficiency, which is why interval-based refresh is used (Liu et al., 1 Jun 2025).

In summary, Low-Rank Informed Sparse Fine-Tuning is a dynamic sparse optimization framework in which low-rank approximation is used to identify Principal Weights in the original model, and only those weights are updated. Within the reported experiments, updating roughly 5% of weights is sufficient to outperform Full FT on several reasoning-oriented evaluations, achieve memory efficiency roughly comparable to LoRA, and reduce catastrophic forgetting relative to both Full FT and LoRA (Liu et al., 1 Jun 2025).