Sens-Merging: Guided LLM Merging

• Sens-Merging is a sensitivity-guided method that computes per-layer coefficients from task-specific sensitivity and cross-task transferability to merge fine-tuned LLM checkpoints.
• It replaces uniform scaling with dynamic, layer-wise adjustments, addressing within-task variability and differential task utility in models like LLaMA2 and Mistral.
• Empirical results show significant improvements in zero-shot accuracy and code generation benchmarks, highlighting its practical advantages over other merging techniques.

Sens-Merging is a sensitivity-guided coefficient adjustment method for merging LLMs that share a common pretrained backbone. It is designed as a plug-and-play enhancement to task vector-based model merging, replacing uniform coefficients with per-layer coefficients derived from two signals: task-specific parameter sensitivity and cross-task transferability. In the formulation introduced in "Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging LLMs" (Liu et al., 18 Feb 2025), the method targets the central limitation of uniform task-vector scaling, namely that parameters and layers contribute unequally within a task and that different task models exhibit unequal utility across tasks.

1. Concept and problem setting

Sens-Merging is situated in the literature on model merging for specialized fine-tuned variants of a common base model. In that setting, several task-specific checkpoints—such as Chat, Math, and Code variants—are merged into a single model by adding fine-tuning deltas to the shared backbone. The paper positions Sens-Merging against Task Arithmetic, TIES-Merging, and DARE, arguing that these methods typically apply uniform coefficients across entire task vectors or whole parameter sets and therefore overlook both within-task variability and across-task variability (Liu et al., 18 Feb 2025).

Within-task variability refers to the fact that parameters and layers contribute unequally to a given task. Across-task variability refers to the fact that some task models transfer more effectively across other tasks, so naive averaging or uniform scaling can introduce interference or negative transfer. Sens-Merging addresses both limitations by computing layer-wise coefficients that modulate task-vector contributions during merging. The method is described as operating at both task-specific and cross-task levels and as remaining compatible with existing merging pipelines rather than replacing them.

The paper studies this mechanism on LLaMA2-7B, LLaMA2-13B, and Mistral-7B families, with fine-tuned variants for general knowledge, mathematical reasoning, and code generation. Evaluation is reported on MMLU, HellaSwag, TruthfulQA, GSM8K, MATH, MBPP, and HumanEval, using zero-shot accuracy and pass@1.

2. Mathematical formulation

Let $\theta$ denote the pretrained base model parameters, and let $\theta_{\text{SFT}}^{(t_k)}$ denote the fine-tuned parameters for task $t_k$. The task vector for task $k$ is

$$\Delta\theta_k = \theta_{\text{SFT}}^{(t_k)} - \theta.$$

A generic per-parameter merging rule is given as

$$\theta_{\text{merge},i} = \theta_i + \sum_{k=1}^K \alpha_{k,i}\,\Delta\theta_{k,i},$$

where $\alpha_{k,i}$ are the merge coefficients. In Sens-Merging, coefficients are constructed layer-wise so that

$\alpha_{k,i} = K \cdot \sigma_k^{\ell(i)},$

with $\ell(i)$ the layer containing parameter $i$, $\theta_{\text{SFT}}^{(t_k)}$0 the number of task vectors, and $\theta_{\text{SFT}}^{(t_k)}$1 the normalized per-task per-layer coefficient.

The final merge rule at layer $\theta_{\text{SFT}}^{(t_k)}$2 is

$\theta_{\text{SFT}}^{(t_k)}$3

Equivalently, the merged model can be written in task-vector form as

$\theta_{\text{SFT}}^{(t_k)}$4

where $\theta_{\text{SFT}}^{(t_k)}$5 is broadcast to the parameters of each layer. The factor $\theta_{\text{SFT}}^{(t_k)}$6 is used for magnitude preservation. The paper emphasizes that Sens-Merging does not optimize an explicit objective over coefficients; the coefficients are computed deterministically from sensitivity and alignment signals rather than by validation-loss minimization or constrained optimization (Liu et al., 18 Feb 2025).

3. Sensitivity and cross-task transferability

The first component of Sens-Merging is task-specific sensitivity. For task $\theta_{\text{SFT}}^{(t_k)}$7 and calibration sample $\theta_{\text{SFT}}^{(t_k)}$8, the parameter-level sensitivity for parameter $\theta_{\text{SFT}}^{(t_k)}$9 in layer $t_k$0 is defined by the first-order loss change under zeroing that parameter:

$t_k$1

Averaging over $t_k$2 calibration instances gives the aggregate per-parameter sensitivity

$t_k$3

Layer-wise sensitivity is then obtained by aggregation over the parameter set $t_k$4 for layer $t_k$5:

$t_k$6

To compare layers within a task, the paper applies L2 normalization across layers:

$t_k$7

The second component is cross-task transferability, computed through logit-space alignment. For calibration samples from task $t_k$8,

$t_k$9

These pairwise quantities are aggregated and normalized as

$k$0

The reported cross-task scaling factors show a consistent ordering in which Math has the largest value, Chat is intermediate, and Code is the most specialized. Specifically, the paper reports for LLaMA2-7B $k$1, $k$2, and $k$3; for Mistral-7B, $k$4, $k$5, and $k$6; and for LLaMA2-13B, $k$7, $k$8, and $k$9 (Liu et al., 18 Feb 2025).

The two signals are combined through a temperature-controlled softmax. For each layer $$\Delta\theta_k = \theta_{\text{SFT}}^{(t_k)} - \theta.$$0,

$$\Delta\theta_k = \theta_{\text{SFT}}^{(t_k)} - \theta.$$1

applied over the $$\Delta\theta_k = \theta_{\text{SFT}}^{(t_k)} - \theta.$$2 tasks for that layer. Lower $$\Delta\theta_k = \theta_{\text{SFT}}^{(t_k)} - \theta.$$3 yields more peaky coefficients, while higher $$\Delta\theta_k = \theta_{\text{SFT}}^{(t_k)} - \theta.$$4 spreads the weights more broadly. The value of $$\Delta\theta_k = \theta_{\text{SFT}}^{(t_k)} - \theta.$$5 is treated as tunable, but the paper does not report a fixed value.

4. Algorithmic pipeline and integration with existing merge methods

Operationally, Sens-Merging first computes task vectors for all fine-tuned checkpoints. It then estimates task-specific sensitivities using calibration samples from each task, typically by computing gradients with respect to the fine-tuned model and aggregating the parameter-level first-order sensitivity scores into layer-wise scores. In parallel, it estimates cross-task transferability by evaluating logit-space alignment on other tasks’ calibration sets. These signals are normalized into $$\Delta\theta_k = \theta_{\text{SFT}}^{(t_k)} - \theta.$$6, which are then used to rescale layer-wise task-vector contributions during merging (Liu et al., 18 Feb 2025).

The paper presents Sens-Merging as a coefficient-selection layer that can be inserted into existing pipelines. For Task Arithmetic, the uniform global scale $$\Delta\theta_k = \theta_{\text{SFT}}^{(t_k)} - \theta.$$7 is replaced by $$\Delta\theta_k = \theta_{\text{SFT}}^{(t_k)} - \theta.$$8 per layer. For TIES-Merging, the method keeps pruning and masking unchanged and substitutes only the scaling stage, so the masked updates $$\Delta\theta_k = \theta_{\text{SFT}}^{(t_k)} - \theta.$$9 are weighted by $$\theta_{\text{merge},i} = \theta_i + \sum_{k=1}^K \alpha_{k,i}\,\Delta\theta_{k,i},$$0. For DARE, the same drop strategy is retained while the uniform scaling factor is replaced by the Sens-Merging coefficients. The method is therefore described as modifying scaling only, rather than conflict resolution or sparsification.

Several practical recommendations are stated. Sensitivity computation can be restricted to fine-tuned delta parameters, such as LoRA weights, for efficiency. Calibration sets are sampled as random subsets from each task’s training set. Existing mask or drop hyperparameters from TIES or DARE are left unchanged. This suggests that Sens-Merging is intended less as a standalone merge rule than as a balancing mechanism for already-established task-vector workflows.

5. Empirical results and ablations

The reported gains are strongest when Sens-Merging is added to Task Arithmetic, though positive results are also reported for TIES-Merging and DARE. The following summary condenses representative average results across the seven reported benchmarks.

Setting	Baseline average	With Sens-Merging
LLaMA2-7B + Task Arithmetic	29.03	34.78
LLaMA2-7B + TIES-Merging	34.73	35.13
LLaMA2-7B + DARE	34.86	35.13
Mistral-7B + Task Arithmetic	37.51	47.85
Mistral-7B + DARE	44.97	48.19
LLaMA2-13B + Task Arithmetic	34.52	40.20

On LLaMA2-7B, Task Arithmetic plus Sens-Merging improves the average from 29.03 to 34.78, with particularly large gains on code generation: MBPP rises from 13.5 to 33.1 and HumanEval from 7.3 to 18.9. TIES-Merging plus Sens-Merging reaches the highest GSM8K score of 47.69 and MATH score of 7.80 among the merged LLaMA2-7B models. On Mistral-7B, Task Arithmetic plus Sens-Merging improves the average from 37.51 to 47.85, including gains of 11.58 on the general domain, 4.86 on math, and 8.45 on code. On LLaMA2-13B, Task Arithmetic plus Sens-Merging improves the average from 34.52 to 40.20, with MBPP increasing from 24.1 to 42.6 and HumanEval from 9.1 to 20.1 (Liu et al., 18 Feb 2025).

A particularly notable result is reported for Mistral-7B with DARE plus Sens-Merging: MBPP reaches 55.1 and HumanEval 43.3, exceeding the specialized fine-tuned Code model scores of 50.9 and 40.9. The paper highlights this as evidence that merged models can outperform specialized fine-tuned models, particularly in code generation tasks.

The ablation study on Task Arithmetic with LLaMA2-7B separates the two scaling components. Adding only task-specific scaling yields an average of 29.41, a small increase over the 29.03 baseline, with improvement concentrated in math, including MATH at +1.38. Adding only cross-task scaling yields 33.40, with strong gains on MMLU, MBPP, and HumanEval but a decline on GSM8K. The full method reaches 34.78, which the paper interprets as a better aggregate balance between specialized and transferable performance. Two-model merges are also reported to benefit from Sens-Merging, and in some cases Chat+Math merges outperform three-model merges under TIES or DARE, indicating that additional task vectors can introduce interference (Liu et al., 18 Feb 2025).

6. Scope, limitations, and terminological context

The paper states several limitations. Sens-Merging is formulated for homogeneous merging, meaning models sharing architecture and backbone. It is primarily validated on LoRA-style fine-tuned models, and the paper notes that behavior may differ for fully fine-tuned models with large weight drift. Sensitivity estimation can be noisy when calibration sets are small, though the method uses L2 normalization across layers and temperature softmax to stabilize coefficients. Calibration selection and the temperature $$\theta_{\text{merge},i} = \theta_i + \sum_{k=1}^K \alpha_{k,i}\,\Delta\theta_{k,i},$$1 can affect results, and optimal choices may be task-dependent. The reported trade-off is explicit: task-specific scaling tends to improve specialized mathematical performance, while cross-task scaling drives aggregate gains, especially in general and code tasks (Liu et al., 18 Feb 2025).

The paper also leaves several reproducibility details unspecified. It does not report wall-clock training or merging times, hardware configurations, or seeds. The additional computational cost is described as modest, consisting mainly of backward passes for sensitivity estimation and forward-only alignment across task pairs. A plausible implication is that the method is best viewed as data-light rather than data-free, because it depends on calibration samples and gradient computation but requires no additional tuning of the merged model.

In the broader nomenclature of model merging, Sens-Merging should not be conflated with several differently named methods that also emphasize sensitivity, activation, semantics, or dynamic routing. "SE-Merging" is a training-free dynamic model merging method based on representation proximity rather than gradient-derived sensitivity (Chen et al., 22 Jun 2025). NeuroMerging decomposes task updates into parallel and orthogonal neuronal subspaces and associates the parallel component with neuron sensitivity, but it is introduced under a different name (Fang et al., 7 Mar 2025). "Activation-Guided Consensus Merging" uses mutual information between activations to derive layer-specific coefficients (Yao et al., 20 May 2025), and "SeMe" is a semantic-based, data-free merging method built around latent semantic alignment (Gu et al., 26 May 2025). Within this landscape, Sens-Merging is specifically the method that combines activation-informed gradient sensitivity with cross-task logit alignment to produce per-layer task-vector coefficients for LLM merging.