Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples (2510.20800v1)

Abstract: Recently, Sharma et al. suggested a method called Layer-SElective-Rank reduction (LASER) which demonstrated that pruning high-order components of carefully chosen LLM's weight matrices can boost downstream accuracy -- without any gradient-based fine-tuning. Yet LASER's exhaustive, per-matrix search (each requiring full-dataset forward passes) makes it impractical for rapid deployment. We demonstrate that this overhead can be removed and find that: (i) Only a small, carefully chosen subset of matrices needs to be inspected -- eliminating the layer-by-layer sweep, (ii) The gradient of each matrix's singular values pinpoints which matrices merit reduction, (iii) Increasing the factorization search space by allowing matrices rows to cluster around multiple subspaces and then decomposing each cluster separately further reduces overfitting on the original training data and further lifts accuracy by up to 24.6 percentage points, and finally, (iv) we discover that evaluating on just 100 samples rather than the full training data -- both for computing the indicative gradients and for measuring the final accuracy -- suffices to further reduce the search time; we explain that as adaptation to downstream tasks is dominated by prompting style, not dataset size. As a result, we show that combining these findings yields a fast and robust adaptation algorithm for downstream tasks. Overall, with a single gradient step on 100 examples and a quick scan of the top candidate layers and factorization techniques, we can adapt LLMs to new datasets -- entirely without fine-tuning.

• The paper presents an adaptation method that uses a single gradient step on 100 examples to efficiently tune large language models.
• The paper utilizes gradient-guided selection and subspace clustering for low-rank compression of weight matrices, reducing overfitting.
• The paper demonstrates up to a 52x speedup and a 24.6% accuracy improvement over traditional fine-tuning methods.

Efficient LLM Adaptation Using a Single Gradient Step

Introduction

The focus of this paper is on a novel method for adapting LLMs to new styles and domains. It emphasizes efficiency by using a single gradient step on merely 100 examples, circumventing the need for fine-tuning. This method addresses the computational challenges associated with domain-specific adaptation of transformer-based LLMs, which are traditionally resource-intensive.

Methodology

The proposed method uses gradient-guided selection of weight matrices within LLMs to perform low-rank compression:

1. Gradient-Based Matrix Selection: Singular value gradients of weight matrices are computed with a single backward pass over 100 examples. This process identifies matrices that warrant low-rank compression to reduce overfitting and align the model with new datasets.

   Figure 1: Efficient LLM adaptation. Singular values across all weight matrices are analyzed for low-rank compression without fine-tuning.

2. Clustering for Decomposition: The rows of selected matrices are clustered into multiple subspaces. Each cluster undergoes independent factorization, capturing heterogeneous structures and minimizing overfitting.
3. Sample Efficiency: The adaptation process relies on only 100 examples, highlighting that model adaptation is dominated by the form of input prompts rather than dataset size.

Experimental Results

The method was evaluated on models like GPT-J and Roberta across several benchmarks, demonstrating significant computation speedups and accuracy improvements:

• Speedups and Accuracy: The approach achieved up to 52 times computational speedup and improved accuracy by up to 24.6 percentage points compared to previous methods such as LASER.

  Figure 2: The accuracy of techniques given computation time for eight datasets while running with GPT-J.

• Comparison to LASER: The proposed method not only outperformed LASER in terms of speed but also maintained or improved accuracy across various datasets. It achieved these improvements through the intelligent selection and compression of matrix weights.

Trade-offs and Implementation

• Computational Requirements: The algorithm significantly reduces the computational load, allowing efficient on-device adaptations of LLMs with limited resources.
• Deployment Strategies: By eliminating the need for fine-tuning, this method facilitates rapid deployment in settings where compute and bandwidth are constrained.

Conclusions

The paper presents a robust, computationally efficient alternative to traditional LLM fine-tuning methods. Through the use of singular value gradients and subspace clustering, it achieves both speed and accuracy. This approach broadens the applicability of LLMs in resource-limited environments and suggests future research directions in exploring more granular matrix adaptations without sacrificing model performance.