Sensitivity-Aware Finetuning

• Sensitivity-aware finetuning is a strategy that identifies which model components most impact task performance and adjusts them selectively based on sensitivity metrics.
• It employs techniques such as Hessian-based rank allocation, gradient-based masking, and representation-space interventions to optimize adaptation capacity.
• Empirical studies show that this method leads to improved robustness and efficiency by preserving pretrained knowledge and mitigating overfitting during domain shifts.

Sensitivity-aware finetuning denotes finetuning procedures that explicitly measure or constrain how strongly model parameters, layers, hidden representations, or outputs affect downstream behavior, and then use that information to decide where adaptation capacity should be placed, how large updates should be, or which sensitivities should be preserved or suppressed. In the literature, this idea appears in several forms: Hessian-based rank allocation for LoRA modules, gradient-based sparse masking, sensitivity-driven expert allocation, representation-space interventions that bound representation drift, noisy training that perturbs low-sensitivity parameters more strongly, and robustness-oriented finetuning that matches Jacobian structure or limits hardware-noise-sensitive layers (Zhang et al., 11 Sep 2025). Taken together, these works treat finetuning not as uniform weight optimization, but as selective control over the model components that matter most for a task, a domain shift, or a deployment constraint (Luo et al., 11 Mar 2026).

1. Sensitivity-aware finetuning as a general paradigm

Sensitivity-aware finetuning starts from the observation that pretrained models are not uniformly reusable. In LLMs, not all parameters or layers are equally important for a downstream task, so adaptation capacity such as LoRA rank can be allocated according to how “sensitive” the loss is to each parameter or weight matrix (Zhang et al., 11 Sep 2025). In continual learning, the same idea is expressed at the representation level: finetuning should explicitly constrain and structure how representations may change, in order to prevent over-sensitivity to new domains and catastrophic forgetting while maintaining enough plasticity to adapt (Luo et al., 11 Mar 2026). In dialogue generation, the issue appears as loss of useful pretrained sensitivities—especially knowledge usage and context sensitivity—during ordinary finetuning, which motivates function-space regularization through mix-review (He et al., 2019).

Across papers, “sensitivity” is therefore not a single quantity. It can mean the effect of perturbing a parameter on the loss, the effect of a feature-space perturbation on classifier decisions, the effect of hardware noise on a layer’s outputs, the effect of representation drift on past-task decision boundaries, or the effect of score-field Jacobians on diffusion trajectories (Nair et al., 2023). This suggests that sensitivity-aware finetuning is best understood as a family of task-adaptive control strategies rather than a single algorithm.

A recurrent misconception is that sensitivity-awareness merely means “tune later layers more.” Several works explicitly contradict that simplification. Sensitivity-LoRA reports that Hessian trace, Top-k curvature, and effective rank emphasize different layer patterns, and states that sensitivity-aware finetuning is not simply “more rank in deeper layers” (Zhang et al., 11 Sep 2025). LoRA-SMoE likewise finds different sensitivity patterns for attention and MLP blocks, rather than a uniform depth heuristic (Xu et al., 6 May 2025).

2. Formalizations of sensitivity

A common formulation treats sensitivity as a first- or second-order effect of perturbing parameters. Sensitivity-LoRA begins from the Taylor expansion

$$E(w + \delta w) = E(w) + g^T \delta w + \frac{1}{2} \delta w^T H \delta w + O(\|\delta w\|^3),$$

with gradient $g$ and Hessian $H$, and then uses the diagonal Hessian entries as weight-level sensitivity scores under the approximation

$$\Delta E \approx \frac{1}{2} \sum_i h_{ii}\,\delta w_i^2.$$

From this, it defines matrix-level global sensitivity as the Hessian trace and supplements it with local metrics such as Top-k curvature and effective rank (Zhang et al., 11 Sep 2025). TsqLoRA instead defines layer sensitivity through gradient-weight products,

$$S_\ell = \mathbb{E}_{(x, y)\sim\mathcal{S}} \Big[ \big| W_\ell \odot \nabla_{W_\ell} \ell(x, y;\theta) \big| \Big],$$

and optionally via Fisher information

$$F_\ell = \mathbb{E}_{(x, y)\sim\mathcal{S}} \big[ \nabla_{W_\ell} \ell(x, y;\theta)^\top \nabla_{W_\ell} \ell(x, y;\theta) \big].$$

These formulations keep sensitivity tied to the current training distribution and to the magnitude of existing weights (Chen et al., 23 Sep 2025).

Other papers define sensitivity in deployment- or architecture-specific ways. SAFT measures layer sensitivity as the standard deviation of the output difference between a clean model and a noisy hardware-simulated model: $S[l] = std(\hat{Y}),$ where $\hat{Y}$ is the difference between noisy and clean layer outputs on the same cached inputs (Nair et al., 2023). Sensi-BERT treats sensitivity as importance of intermediate dimensions in MHSA and FFN modules, represented by learned masks trained with cross-entropy plus an $\ell_1$ penalty; higher mask values indicate dimensions that should survive budgeted trimming (Kundu et al., 2023). PATS uses the pruning-style proxy

$$s_j \approx |\theta_j \nabla_{\theta_j}\mathcal{L}(\boldsymbol{\Theta})|,$$

and then perturbs low-sensitivity parameters more strongly during finetuning (Zhang et al., 2022). GEM redefines sensitivity in relative, scale-aware terms through the Gradient-to-Weight Ratio,

$g$0

arguing that relative change, not absolute change, better captures effective downstream adaptation (Kang et al., 22 Aug 2025).

Sensitivity can also be defined in feature or representation space. CoRe intervenes directly on hidden representations

$g$1

and derives the stability bound

$g$2

showing that the geometry of the low-rank subspace controls how much representations can drift during finetuning (Luo et al., 11 Mar 2026). IGSM defines sensitivity through the Jacobian $g$3 of a diffusion score model and matches second-order directional expansions

$g$4

thereby aligning local geometric and temporal dynamics between dense and pruned diffusion models (Zheng et al., 3 Jun 2025).

3. Mechanisms for using sensitivity during finetuning

Once sensitivity is estimated, methods differ in how they convert it into adaptation decisions. A large class of methods uses sensitivity to allocate trainable capacity. Sensitivity-LoRA computes global and local Hessian-based scores, combines them into a final matrix score $g$5, and then assigns matrix-specific LoRA ranks by Scaled Rank Allocation: $g$6 The total parameter budget is fixed; sensitivity only redistributes rank across layers and matrices (Zhang et al., 11 Sep 2025). TsqLoRA follows the same general logic with dynamic, layer-wise ranks: $g$7 where sensitivity is computed on a quality-filtered subset $g$8, and the effective budget is modulated by average data quality (Chen et al., 23 Sep 2025). LoRA-SMoE uses sensitivity to decide expert multiplicity rather than rank, allocating experts only to the most sensitive parameter blocks under a budget, with separate strategies for unified, module-aware, or per-matrix allocation (Xu et al., 6 May 2025).

Another class uses sensitivity for sparse or selective weight updates. GEM computes per-parameter GWR scores, then forms a layer-wise distribution

$g$9

its entropy

$H$0

and a layer importance score

$H$1

Layer budgets are then set as

$H$2

and masked training updates only the selected parameters (Kang et al., 22 Aug 2025). Sensi-BERT similarly learns sensitivity masks over intermediate dimensions and then threshold them to satisfy a global budget $H$3, producing a trimmed model that is subsequently finetuned (Kundu et al., 2023). SPT applies a related principle in vision PEFT: it first ranks individual weights by squared gradients, then either tunes them directly via binary masks or replaces heavily populated sensitive matrices with structured modules such as LoRA or Adapters, depending on whether the count of sensitive entries exceeds a threshold (He et al., 2023).

A third mechanism is sensitivity-aware regularization rather than selection. PATS adds Gaussian noise to parameters during finetuning, with variance inversely related to sensitivity: $H$4 so low-sensitivity parameters are perturbed more strongly and high-sensitivity ones are protected (Zhang et al., 2022). SAFT freezes low-sensitivity layers and updates only hardware-noise-sensitive layers during noise-injection training, achieving comparable accuracy to full noise-injection training with lower cost (Nair et al., 2023). Mix-review regularizes finetuning in function space by mixing pretraining data with target data,

$H$5

which preserves context sensitivity and pretrained knowledge better than standard finetuning (He et al., 2019).

4. Beyond parameter saliency: representation, input, and deployment sensitivity

Sensitivity-aware finetuning is not restricted to choosing important parameters. CoRe shifts finetuning from weight space to representation space, inserting low-rank intervention modules into ViT hidden states and optimizing a representation-level objective

$H$6

The first term enforces representation alignment; the second keeps the intervention basis near orthonormal, which makes the bound on representation drift explicit (Luo et al., 11 Mar 2026). This suggests a broader interpretation of sensitivity-awareness: not merely finding parameters with high loss influence, but explicitly deciding which representational directions may change and by how much.

Robustness-oriented works extend the same idea to input sensitivity. In diffusion-model finetuning, IGSM argues that standard denoising score matching supervises pointwise scores at each timestep but does not control how perturbations evolve along the denoising trajectory. Its second-order Jacobian matching term transfers the teacher’s local expansion and contraction behavior to the pruned student, making finetuning explicitly aware of geometric and temporal sensitivity (Zheng et al., 3 Jun 2025). In semantic segmentation, adaptive sensitivity-guided augmentation defines sensitivity as

$H$7

where $H$8 is model accuracy under corruption and $H$9 measures distributional shift in feature space; the resulting augmentation policy is recomputed online to focus training on corruption intensities where performance drops rapidly (Zheng et al., 2024).

Classifier-level sensitivity can itself be the target of finetuning. Work on adversarial robustness distinguishes the high sensitivity of FC classifiers from the lower sensitivity of $$\Delta E \approx \frac{1}{2} \sum_i h_{ii}\,\delta w_i^2.$$0-distance-based classifiers, and introduces an HPM $$\Delta E \approx \frac{1}{2} \sum_i h_{ii}\,\delta w_i^2.$$1-reclassifier that mixes stable dataset-level prototypes with dynamic batch-level prototypes, then finetunes this lightweight module with minimal updates to improve adversarial robustness (Wang, 1 Jun 2026). That paper explicitly frames sensitivity as a trade-off: high sensitivity improves discriminability but increases vulnerability, while low sensitivity improves robustness but can limit performance.

Deployment constraints create further variants. SAFT measures the effect of quantization and analog noise at the layer level and finetunes only the most noise-sensitive layers (Nair et al., 2023). In transferred financial sentiment models, sensitivity appears as hypersensitivity to stochasticity in learning rate, batch size, and layer freezing, especially for BERT; freezing earlier layers is recommended because they contain essential word pattern information that should be maintained (Qian et al., 2022). In bias mitigation, “overwriting pretrained bias with finetuning data” shows that relatively minor interventions to the finetuning data distribution can overwrite inherited bias from pretraining, making the finetuning stage explicitly sensitive to protected or underrepresented groups (Wang et al., 2023).

5. Empirical patterns across domains

Across domains, sensitivity-aware finetuning consistently reports gains under tight budgets or difficult constraints. On GLUE with RoBERTa-base, Sensitivity-LoRA reports an average score of 85.94, compared with 85.06 for uniform LoRA, 85.20 for AdaLoRA, and 85.19 for DyLoRA, and it performs best or tied-best across multiple tasks (Zhang et al., 11 Sep 2025). On generative tasks with Qwen2.5-7B and LLaMA3.1-8B, the same method improves average BLEU-4/ROUGE metrics over LoRA and adaptive-rank baselines, while using a one-off sensitivity computation that can be completed in tens of seconds on an 8B model (Zhang et al., 11 Sep 2025). TsqLoRA likewise shows that sensitivity-aware rank allocation combined with data-quality sampling outperforms standard LoRA, AdaLoRA, and ElaLoRA under matched parameter budgets on GLUE and XSum (Chen et al., 23 Sep 2025).

Sparse and mask-based methods show similar effects. GEM reports that with OPT-125M, OPT-1.3B, and Phi-2, tuning only 0.1% of parameters via GWR plus entropy-guided masking can match or exceed full finetuning on average across GLUE and SuperGLUE-style tasks, and can outperform other PEFT baselines at the same budget (Kang et al., 22 Aug 2025). Sensi-BERT reports that BERT-base can be reduced to 53.3M parameters at budget $$\Delta E \approx \frac{1}{2} \sum_i h_{ii}\,\delta w_i^2.$$2 while staying competitive with or surpassing several 66–68M-parameter baselines on MNLI, QNLI, QQP, and SST-2, without teacher-based distillation (Kundu et al., 2023). SPT shows that, on FGVC and VTAB-1k, sensitivity-aware placement of Adapters or LoRA modules in ViT-B/16 outperforms fixed-placement PEFT baselines, and under self-supervised pretraining it can outperform full finetuning by large margins (He et al., 2023).

Sensitivity-awareness is also effective in multi-task or continual settings. LoRA-SMoE reports that LoRA-SMoE-S(60%) achieves the best average accuracy across eight commonsense QA tasks while using fewer trainable parameters than HydraLoRA (Xu et al., 6 May 2025). CoRe reports higher Avg and Last accuracy than adapters, prompts, SSF, and full finetuning in TIL, DIL, and CIL settings with ViT-B/16, linking these gains to reduced representation drift and better stability–plasticity balance (Luo et al., 11 Mar 2026). Sens-Merging extends the same logic beyond finetuning into post-hoc model composition, showing that sensitivity-guided, layer-wise task coefficients improve merged model performance over uniform task-vector scaling across chat, math, and code tasks (Liu et al., 18 Feb 2025).

In robustness and systems settings, sensitivity-aware methods often trade minimal extra analysis for substantial savings or robustness improvements. SAFT achieves comparable accuracy to full noise-injection training while being 2x to 8x faster by freezing low-sensitivity layers (Nair et al., 2023). IGSM substantially narrows the FID gap between pruned and dense diffusion models across CIFAR-10, CelebA, LSUN-Church, and LSUN-Bedroom by matching second-order Jacobian projections (Zheng et al., 3 Jun 2025). PATS consistently improves BERT-base and RoBERTa-large on GLUE, and the authors report that better-performing models exhibit more concentrated distributions of parameter sensitivities (Zhang et al., 2022). In ASR, sensitivity-guided one-shot pruning shows that decoder FFNs are pruning-fragile, whereas decoder self-attention and the last encoder layers contain redundancy that can improve WER when removed, which reframes pruning as a sensitivity-aware regularizer (Irigoyen et al., 11 Nov 2025).

6. Limitations, tensions, and future directions

A central tension is that sensitivity is task-dependent, architecture-dependent, and sometimes objective-dependent. Gradient-based measures are cheap but first-order; Fisher and Hessian-based measures may better capture curvature but add complexity or approximation error (Chen et al., 23 Sep 2025). Sensitivity defined on pretrained weights may not stay fixed throughout long finetuning, although several papers report surprising stability of rankings across calibration sets, domains, and training epochs (Zhang et al., 11 Sep 2025). This suggests robustness of the signal, but not universality.

Another limitation is that sensitivity can be beneficial or harmful depending on what is being measured. Dialogue work shows that standard finetuning can erase useful context sensitivity and knowledge, so preserving pretrained sensitivity may be desirable (He et al., 2019). Adversarial-robustness work instead treats high classifier sensitivity as a liability that should be reduced or restructured (Wang, 1 Jun 2026). Bias-mitigation work shows that inherited sensitivities to spurious correlations can be corrected through curated finetuning data (Wang et al., 2023). This suggests that sensitivity-aware finetuning is not only about maximizing sensitivity at important locations, but about distinguishing desirable from undesirable sensitivities.

A further methodological issue is granularity. Some methods allocate rank or experts at the matrix or block level (Zhang et al., 11 Sep 2025, Xu et al., 6 May 2025); others select scalar weights (Kang et al., 22 Aug 2025), dimensions (Kundu et al., 2023), or hidden-state subspaces (Luo et al., 11 Mar 2026). No single granularity dominates across all applications. A plausible implication is that the appropriate sensitivity unit depends on the adaptation mechanism itself: low-rank adapters call for matrix-level scores, sparse finetuning favors scalar or channel-level scores, and continual learning may be better controlled in representation space.

Several directions recur across the literature. These include extending sensitivity-aware allocation to other adapter families and modalities, integrating task-aware sensitivity with data selection, designing dynamic rather than one-shot sensitivity refresh, and developing stronger theoretical links between sensitivity scores and downstream generalization (Zhang et al., 11 Sep 2025). More broadly, the surveyed work indicates that finetuning quality increasingly depends on identifying where a model is plastic, where it is brittle, and where its pretrained inductive structure should be preserved rather than overwritten.