TsqLoRA: Sensitivity & Quality Adaptation

• TsqLoRA is a parameter-efficient fine-tuning method that combines quality-aware data sampling with sensitivity-aware dynamic rank allocation.
• It computes a sample quality score using gradient norms and loss reduction to select the most informative training examples.
• The method dynamically assigns layer-specific low-rank capacities based on gradient sensitivity, improving performance under fixed rank budgets.

TsqLoRA, short for Towards Sensitivity and Quality Low-Rank Adaptation, is a parameter-efficient fine-tuning method for large pre-trained models that jointly optimizes which data to train on and how much low-rank capacity to allocate to each layer. It was introduced to address two limitations that the method identifies in existing PEFT practice: the tendency to use a uniform training set despite variation in sample informativeness, and the tendency to use static or only parameter-side rank assignment despite variation in layer sensitivity. TsqLoRA combines a quality-aware sampling mechanism for selecting informative training examples with a sensitivity-aware dynamic rank allocation module that redistributes LoRA capacity across layers under a fixed rank budget, while keeping the backbone frozen and updating only LoRA parameters (Chen et al., 23 Sep 2025).

1. Conceptual basis and problem setting

TsqLoRA is situated in the PEFT lineage that includes Adapters, Prefix-Tuning, P-Tuning, and LoRA. In the formulation adopted by the method, full fine-tuning of large PLMs such as BERT, GPT, DeBERTa, and BART is costly because it updates all parameters, requires large memory for optimizer states and activations, and is often infeasible or inefficient in resource-constrained settings. LoRA addresses this by adding trainable low-rank matrices $A,B$ to selected weight matrices and training only those matrices. TsqLoRA retains the standard LoRA backbone but argues that PEFT methods commonly overlook two sources of inefficiency: layer sensitivity and training data quality (Chen et al., 23 Sep 2025).

The first premise is that different layers contribute differently to downstream performance. Some layers benefit greatly from extra adaptation capacity, whereas others can remain almost unchanged; a fixed uniform rank is therefore suboptimal. The second premise is that not all training examples are equally informative. Noisy, redundant, or easy samples may dominate optimization and waste gradient steps, whereas recent data-selection work such as TS-DSHAPLEY, 0.5% Data, and LESS suggests that careful sample selection can match or outperform full-data fine-tuning. TsqLoRA is defined precisely as the combination of these two ideas: it scores each sample $(x_i,y_i)$ by a quality score $Q(x_i)$, constructs a weighted subset and sampling distribution from those scores, estimates layer sensitivity $S_\ell$ from gradients, and dynamically allocates layer-wise LoRA ranks under a base budget $R_0$ (Chen et al., 23 Sep 2025).

In this sense, TsqLoRA is a fine-tuning-time method rather than a fusion method. This distinction is useful because contemporaneous LoRA work also includes query-adaptive fusion methods such as qa-FLoRA, which computes layer-level fusion weights across pre-existing adapters without additional training data, and tensorized or clustered merging methods such as TensLoRA and TC-LoRA, which address different structural or multi-task interference problems (Shukla et al., 12 Dec 2025, Marmoret et al., 22 Sep 2025, Su et al., 6 Aug 2025).

2. Quality-aware sampling mechanism

The data-selection side of TsqLoRA is organized around a sample-quality score defined as a weighted combination of three terms:

$$Q(x_i) = \alpha_1 \left\|\nabla_\theta \ell(x_i, y_i;\theta)\right\|_2 + \alpha_2 \Delta \ell(x_i) + \alpha_3 \Delta C(x_i).$$

Here, $\ell(x_i,y_i;\theta)$ is the loss on sample $x_i$, $\nabla_\theta \ell(x_i,y_i;\theta)$ is its gradient with respect to model parameters, and $\alpha_1,\alpha_2,\alpha_3$ are tunable weights. The first term is the gradient norm, interpreted as a signal of informativeness or difficulty. The second is the potential loss reduction,

$(x_i,y_i)$0

which approximates how much a hypothetical one-step update on that sample would reduce loss. The third term, $(x_i,y_i)$1, is described as an abstract contribution to convergence speed, for example through alignment with the global gradient or reduction in variance; the method explicitly notes that it is conceptual rather than given in closed form (Chen et al., 23 Sep 2025).

After computing $(x_i,y_i)$2, typically in a warm-up pass, TsqLoRA defines a softmax sampling distribution

$(x_i,y_i)$3

with temperature $(x_i,y_i)$4 controlling sharpness. Small $(x_i,y_i)$5 produces a peaky distribution that emphasizes high-quality samples, while large $(x_i,y_i)$6 approaches uniform sampling. A weighted subset $(x_i,y_i)$7 is then constructed from these probabilities, and mini-batches are sampled from $(x_i,y_i)$8 with probabilities proportional to $(x_i,y_i)$9. The method characterizes this as a quality-aware curriculum that concentrates optimization on informative samples and reduces redundancy and noise (Chen et al., 23 Sep 2025).

The experimental interpretation of this mechanism is supported by several figure-level observations. Figure 1 shows that, across GLUE, TsqLoRA uses fewer train samples and trains faster than ElaLoRA while achieving better or comparable performance. Figure 2, on QNLI, reports that loss, gradient contribution, and quality scores exhibit similar Gaussian-like distributions, which the authors interpret as evidence that $Q(x_i)$0 captures relevant training signals while also making the choice of emphasized regions in the distribution consequential (Chen et al., 23 Sep 2025).

3. Sensitivity-aware dynamic rank allocation

The second component of TsqLoRA is a layer-wise rank scheduler driven by sensitivity estimates over the current quality-selected subset. For a layer $Q(x_i)$1 with weight matrix $Q(x_i)$2, the default sensitivity score is

$Q(x_i)$3

where $Q(x_i)$4 denotes element-wise multiplication and the expectation is over the current subset $Q(x_i)$5. This is a grad–weight product metric: large magnitude suggests that a layer contains parameters with both substantial values and strong update signals. An alternative Fisher information–based sensitivity is also given,

$Q(x_i)$6

The resulting sensitivity scores are normalized to $Q(x_i)$7 to enable stable comparison across layers and to permit direct conversion into rank proportions (Chen et al., 23 Sep 2025).

TsqLoRA retains the standard LoRA perturbation form

$Q(x_i)$8

or optionally $Q(x_i)$9, but replaces the uniform rank $S_\ell$0 with a dynamic layer-specific rank $S_\ell$1. For each adapted layer $S_\ell$2,

$S_\ell$3

The allocation rule at training iteration $S_\ell$4 is

$S_\ell$5

where $S_\ell$6 is the base rank budget and $S_\ell$7 is a global scaling factor proportional to the average data quality in the current subset. Relative layer sensitivity determines how the budget is divided; average data quality modulates the effective capacity globally. In practice, ranks are discretized, bounded by minimum and maximum constraints, and LoRA matrices are pruned or expanded accordingly through masking or the addition of new dimensions (Chen et al., 23 Sep 2025).

This formulation places TsqLoRA in direct relation to rank-adaptive LoRA variants such as AdaLoRA and ElaLoRA. The distinction claimed by TsqLoRA is that its rank distribution is tied not only to parameter-side importance but also to the quality of the active training subset. Figure 3 on QNLI reports that layers with higher sensitivity receive higher final ranks, which is presented as a validation of the allocation mechanism (Chen et al., 23 Sep 2025).

4. Training algorithm and implementation characteristics

The method is presented as a five-stage procedure. First, LoRA modules are inserted into selected layers such as attention and feedforward projections with a base rank $S_\ell$8, and moving statistics for sensitivity are initialized. The hyperparameters $S_\ell$9, $R_0$0, $R_0$1, $R_0$2, and $R_0$3 are set. Second, a short warm-up pass over the training pool $R_0$4 computes per-sample losses and gradients, from which $R_0$5 and the sampling distribution $R_0$6 are derived. Third, the main training loop samples mini-batches from the quality-selected subset $R_0$7, computes loss and backpropagation, updates only LoRA parameters $R_0$8, and accumulates layer sensitivity statistics such as grad–weight products or Fisher terms. Fourth, at scheduled iterations $R_0$9, the method recomputes normalized sensitivities, estimates average data quality, reallocates ranks according to the formula above, prunes or expands LoRA matrices, and may optionally refresh $$Q(x_i) = \alpha_1 \left\|\nabla_\theta \ell(x_i, y_i;\theta)\right\|_2 + \alpha_2 \Delta \ell(x_i) + \alpha_3 \Delta C(x_i).$$0 through a light pass over $$Q(x_i) = \alpha_1 \left\|\nabla_\theta \ell(x_i, y_i;\theta)\right\|_2 + \alpha_2 \Delta \ell(x_i) + \alpha_3 \Delta C(x_i).$$1. Fifth, it returns the fine-tuned model $$Q(x_i) = \alpha_1 \left\|\nabla_\theta \ell(x_i, y_i;\theta)\right\|_2 + \alpha_2 \Delta \ell(x_i) + \alpha_3 \Delta C(x_i).$$2 with adapted LoRA modules (Chen et al., 23 Sep 2025).

Several implementation features are emphasized. TsqLoRA is described as plug-and-play with LoRA because it uses standard LoRA modules and adds only two auxiliary mechanisms: data-quality scoring with weighted sampling, and sensitivity estimation with rank adjustment logic. No architectural modifications to the backbone are required; the paper explicitly states compatibility with backbones such as DeBERTa and BART. For GLUE, the reported setting is batch size 64, 50 epochs, and $$Q(x_i) = \alpha_1 \left\|\nabla_\theta \ell(x_i, y_i;\theta)\right\|_2 + \alpha_2 \Delta \ell(x_i) + \alpha_3 \Delta C(x_i).$$3. For XSum, the experiments use BART-base, 10 epochs, Adam, and a linear learning-rate schedule, matching ElaLoRA’s setup. Learning rates and other optimizer parameters follow the baseline LoRA/ElaLoRA settings (Chen et al., 23 Sep 2025).

This implementation profile also clarifies what TsqLoRA is not. It is not a tensorized reformulation of LoRA in the sense of TensLoRA, which aggregates updates into higher-order tensors and factorizes them with Tucker decomposition, nor is it a query-time multi-adapter merging framework such as qa-FLoRA. A plausible implication is that TsqLoRA occupies a relatively conservative design point in the LoRA design space: it changes sampling policy and layer-wise capacity allocation while preserving the standard LoRA module interface (Marmoret et al., 22 Sep 2025, Shukla et al., 12 Dec 2025).

5. Experimental setup and reported results

The empirical study evaluates TsqLoRA on GLUE with DeBERTa-v3-base and on XSum with BART-base. GLUE comprises MNLI, QNLI, SST-2, RTE, CoLA, QQP, MRPC, and STS-B, using Accuracy for MNLI, QNLI, SST-2, RTE, CoLA, and MRPC, Accuracy/F1 for QQP and MRPC, and Pearson correlation for STS-B. XSum is evaluated with ROUGE-1/2/L. Baselines include LoRA, AdaLoRA, ElaLoRA, and, on GLUE subsets, specialized data-selection methods such as TS-DSHAPLEY, 0.5% Data, AlpaGasus, LESS, LIMA, and Instruction Mining (Chen et al., 23 Sep 2025).

The main GLUE results are reported for ranks $$Q(x_i) = \alpha_1 \left\|\nabla_\theta \ell(x_i, y_i;\theta)\right\|_2 + \alpha_2 \Delta \ell(x_i) + \alpha_3 \Delta C(x_i).$$4. At $$Q(x_i) = \alpha_1 \left\|\nabla_\theta \ell(x_i, y_i;\theta)\right\|_2 + \alpha_2 \Delta \ell(x_i) + \alpha_3 \Delta C(x_i).$$5 with approximately 0.33M parameters, TsqLoRA obtains an average “All” score of 88.09, compared with 87.51 for LoRA, 87.39 for AdaLoRA, and 88.01 for ElaLoRA. At $$Q(x_i) = \alpha_1 \left\|\nabla_\theta \ell(x_i, y_i;\theta)\right\|_2 + \alpha_2 \Delta \ell(x_i) + \alpha_3 \Delta C(x_i).$$6 with approximately 0.66M parameters, TsqLoRA reaches 88.72, against 87.63 for LoRA, 87.89 for AdaLoRA, and 88.72 for ElaLoRA; task-specific figures highlighted in the paper include QQP 91.45/88.81 and RTE 87.36. At $$Q(x_i) = \alpha_1 \left\|\nabla_\theta \ell(x_i, y_i;\theta)\right\|_2 + \alpha_2 \Delta \ell(x_i) + \alpha_3 \Delta C(x_i).$$7 with approximately 1.66M parameters, TsqLoRA attains 88.86, slightly above ElaLoRA’s 88.84, and records the best SST-2 accuracy at 96.10 (Chen et al., 23 Sep 2025).

On GLUE subset comparisons against data-selection methods, TsqLoRA is reported at 96.00 accuracy on SST-2, matching Instruction Mining and exceeding TS-DSHAPLEY (95.30) and AlpaGasus (95.90). On QQP it reports 92.00 accuracy, compared with 91.90 for TS-DSHAPLEY, 90.92 for LIMA, and 91.02 for Instruction Mining. On RTE it reports 86.50, compared with 80.10 for TS-DSHAPLEY, 86.34 for AlpaGasus, and 85.42 for LIMA. The paper interprets this as evidence that combining PEFT with data-quality-aware sampling can be competitive with specialized data-selection methods (Chen et al., 23 Sep 2025).

On XSum, full fine-tuning with 124.65M parameters achieves 40.61 / 17.76 / 32.91 for ROUGE-1/2/L. Under PEFT budgets, TsqLoRA at rank $$Q(x_i) = \alpha_1 \left\|\nabla_\theta \ell(x_i, y_i;\theta)\right\|_2 + \alpha_2 \Delta \ell(x_i) + \alpha_3 \Delta C(x_i).$$8, 0.41M parameters reports 37.29 / 14.82 / 29.76, compared with 36.61 / 14.28 / 29.23 for LoRA, 36.97 / 14.42 / 29.42 for AdaLoRA, and 37.24 / 14.66 / 29.76 for ElaLoRA. At rank $$Q(x_i) = \alpha_1 \left\|\nabla_\theta \ell(x_i, y_i;\theta)\right\|_2 + \alpha_2 \Delta \ell(x_i) + \alpha_3 \Delta C(x_i).$$9, 1.22M parameters, TsqLoRA gives 38.00 / 15.30 / 30.53, matching the best ROUGE-1 and ROUGE-2 and achieving the best ROUGE-L against 37.64 / 15.30 / 30.19 for LoRA, 37.80 / 15.18 / 30.28 for AdaLoRA, and 38.00 / 15.30 / 30.42 for ElaLoRA (Chen et al., 23 Sep 2025).

Ablation results isolate the contribution of sensitivity-aware rank allocation. On XSum, removing sensitivity-aware allocation while keeping the same data subset reduces performance from 37.29 / 14.82 / 29.76 to 36.61 / 14.28 / 29.23 at $\ell(x_i,y_i;\theta)$0, and from 38.00 / 15.30 / 30.53 to 37.64 / 15.18 / 30.19 at $\ell(x_i,y_i;\theta)$1. The paper reports similar gains on RTE and MRPC, concluding that dynamic rank allocation based on sensitivity is particularly important in low-rank regimes (Chen et al., 23 Sep 2025).

6. Interpretation, limitations, and position in the LoRA ecosystem

The principal contribution claimed for TsqLoRA is the joint consideration of data and parameters. Most PEFT methods optimize the parameter side only; TsqLoRA combines data-quality-aware sampling and sensitivity-aware rank allocation in a single framework. The reported strengths are efficiency in low-rank and low-data regimes, improved or matched performance relative to LoRA-family baselines at the same parameter budget, reduced effective training data usage, and plug-and-play compatibility with existing LoRA implementations including Hugging Face-style workflows (Chen et al., 23 Sep 2025).

The method also has explicit limitations. The warm-up pass for computing $\ell(x_i,y_i;\theta)$2 and the periodic sensitivity updates require extra forward and backward passes and additional storage. For very large datasets, gradient norms and one-step loss-reduction estimates for all samples may be expensive, although the paper notes that they can be approximated or subsampled. Performance depends on the hyperparameters $\ell(x_i,y_i;\theta)$3, and poor tuning may yield only marginal improvement over simpler baselines. The quality metric assumes that high gradient norm and high potential loss reduction correlate with informativeness, which may fail under noisy labels or adversarial examples. Dynamic pruning and expansion of LoRA ranks is also more complex than fixed-rank LoRA because optimizer states and initialization of new dimensions must be handled carefully (Chen et al., 23 Sep 2025).

Within the broader LoRA literature, TsqLoRA is best regarded as a specialized PEFT algorithm rather than a general structural reformulation of LoRA. qa-FLoRA addresses data-and-training-free query-adaptive fusion of LoRAs at inference time, not fine-tuning-time sample selection (Shukla et al., 12 Dec 2025). ROMA addresses QLoRA-based on-device acceleration with ROM for the quantized base model and SRAM for LoRA weights and KV cache, a deployment-level concern rather than a data-selection or rank-allocation method (Wang et al., 17 Mar 2025). IR-QLoRA focuses on information retention in quantized LoRA finetuning through Information Calibration Quantization and Information Elastic Connection (Qin et al., 2024). TensLoRA and TC-LoRA investigate tensorization and clustered merging, respectively, rather than sensitivity-aware rank scheduling during task adaptation (Marmoret et al., 22 Sep 2025, Su et al., 6 Aug 2025). A later unified review of LoRA variants reports that TsqLoRA is not included in its taxonomy, LoRAFactory description, or empirical tables, which suggests that the method had not yet been absorbed into that particular standardized survey framework (He et al., 30 Jan 2026).

The future directions stated for TsqLoRA include more fine-grained sensitivity measures such as low-rank Hessian approximations and layer-wise Fisher diagonal, extensions to multimodal models and other architectures, and alternative quality metrics such as Shapley values, mutual information, or more principled convergence predictors (Chen et al., 23 Sep 2025). This suggests that TsqLoRA is less a terminal variant than a template for coupling data valuation with capacity allocation inside PEFT.